abysmal NTFS write speeds USB drive

General support questions
Post Reply
danielrg76
Posts: 5
Joined: 2020/07/11 09:04:18

abysmal NTFS write speeds USB drive

Post by danielrg76 » 2020/07/11 09:13:34

OS Version: CentOS Linux release 7.4.1708

I am running CentOS on an old laptop, using it as a Plex server. My Plex library is carried on a 2TB Toshiba USB drive formatted NTFS. I have a second 4TB Toshiba USB with NTFS also mounted which I rsync the library to nightly as a backup. The NTFS USB drives are using the ntfs-3g driver.

I rip my media using a windows machine on the network, then use winSCP from windows to the Linux machine to copy new files onto the Plex drive.

About two weeks ago, I copied over our photo collection, consisting of 50000 photos and videos, ~180 GB to both the 2TB and 4TB drives.

I used to consistently get 10MB/s to 11 MB/s (I have a 100Mbps router in between so this is expected max) copying using WinSCP to my Linux Plex drive(s).

About 20% into copying the photo library, my transfer speed started to go all over the map, then just slowed to a crawl of 60KB/s to 600 KB/s, and it couldn’t seem to pull out of it. I also start to get host comms errors. This NEVER happened before, and I haven't changed anything on this laptop except the plex server version. Oh, and putting all these new files on...

Now, all my transfers to these drives start fast when I first start the WinSCP copy, but become abysmally slow after about 2-3 minutes, hovering around 20-100 KB/s if it's a big transfer of several GB (which is common if building a media library).

If I copy to the boot drive, I get my 11 MB/s speeds back and they stay up fine. I even tried copying to the boot drive then rsync to the PLEX USB NTFS drive, and even that method gets abysmal speeds. So it really seems to be some problem with the drives or the ntfs-3g driver. But I haven’t changed anything with regards to this setup except putting the photos on there.

Something is up but I can’t figure it out. I have tried all sorts of mount options. Currently this is my mount line in /etc/fstab:

UUID=34A060C3A0608CE4 /mnt/plex01 ntfs-3g async,noatime,big_writes 0 0

I’ve tried ‘defaults’ as the mount options, I’ve tried using the UUID as device, using /dev/sdb, nothing seems to help the problem.

One thing I haven’t tried is connecting the USB drives to a windows machine and running chkdsk or defragging (or maybe there's a better linux utility out there but I worry about the SPEED :-) ).

Maybe something got screwed up when I started copying over the 50000 file photo library? That does seem to be when this all started. Any other suggestions?

Also, when I abort the transfer due to low speeds, and then unmount the drive, it can take up to several minutes to unmount. I wonder if the speeds slow to abysmal when some software buffer fills up because of the slow disk, then when I umount it is still writing to disk from buffer hence the long umount time? Could it be my USB settings somehow are messed up? I checked the USB port versions. Port is USB 2.0, drive is USB 2.1. /proc/.../speed is 480.

Thanks for any ideas on what to try. the ntfs-3g support forums seem old and abandoned...

User avatar
TrevorH
Forum Moderator
Posts: 29069
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: abysmal NTFS write speeds USB drive

Post by TrevorH » 2020/07/11 16:34:41

Start by running yum update to get yourself up to date. CentOS 7.4 has been unsupported since the release of 7.5 so your system is currently over 3 years out of date and susceptible to any number of high severity security vulnerabilities. Running yum update will get you current and should eb something you run on a regular basis anyway, just like running Windows Update.

Only the current version of CentOS gets any support or fixes.

It is normal for a copy to start fast and then slow down. This is because when you first start the copy, linux caches the writes in RAM so that bit will be really fast. Once it's filled up all available free RAM cache with data, the copy will then slow down as it now has to go at the speed that the device can actually write at. That's also why it takes a while to umount since the same phenomenon is still in play and there are potentially multi-GBs of data cached in RAM that need to be written.
I also start to get host comms errors.
Say more about that as this sounds like the real problem. Show us the exact errors you get. Also post the output from lsusb.py so we can see info about the USB connection and the devices in use. I'd also recommend running smartctl -a /dev/sdX (part of the smartmontools package) against the drive in question and posting that so we can see the health of your device. You may need to specify -d sat or some other parameter from the smartctl man page to get it to talk ot a USB device.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

danielrg76
Posts: 5
Joined: 2020/07/11 09:04:18

Re: abysmal NTFS write speeds USB drive

Post by danielrg76 » 2020/07/12 03:54:14

I have an OLD machine running my plex server - it's a 32-bit CPU. I found a 32-bit build of CentOS 7 and used that on this machine.

I am doing the update, but I am hoping it will work... it can only download i686 packages which may not be complete or compatible with this special build. If it breaks everything... well maybe it was time. I have an old 64-bit laptop sitting around now that I didn't have when I made this plex server. So if it breaks, I should probably convert it into my Plex server now, and get rid of this old 32-bit guy... it's just that it just keeps on going!

I'll do the yum upgrade and go from there... if everything breaks maybe once I get plex running on this other 64-bit machine I won't even have to deal with this particular problem anymore :-)

danielrg76
Posts: 5
Joined: 2020/07/11 09:04:18

Re: abysmal NTFS write speeds USB drive

Post by danielrg76 » 2020/07/12 06:37:11

... yeah, my computer is dead now after the yum update, as I suspected might happen. I can't boot off any of the startup options anymore. they all kick me to emergency mode where I don't have network or anything for doing a yum undo or anything like that. If I try to start the network service (or any other service for that matter), I get some error about "authorization not available check if polkit service is running" Greek to me. And searching these forums looks like some nasty fixes and I suspect it wouldn't be enough. journald shows all sorts of errors on bootup too now.

I backed up my data before so nothing critical is lost except time - I didn't bother backing up my boot drive, since this is such an old computer I figured if this didn't work I'd let it go the way of all the earth.

Probably time to build a plex server from something other than ancient hardware.

The real backlash on this will be telling my wife our Plex server is dead until I get a new one up and running, probably a couple weeks out :?

BTW, I did look at my NTFS drives using a defrag software on Windows. the ntfs-3g driver had written the files in what looked like a very odd way, peppering them all over the disk. It might have been the way it wrote data to the disk that made them slow for writing, since that had been the only real change was adding those 50000 photos to the drives.

Running SMART tools on the drives show they are in good health (they are fairly new).

danielrg76
Posts: 5
Joined: 2020/07/11 09:04:18

Re: abysmal NTFS write speeds USB drive

Post by danielrg76 » 2020/07/13 02:55:25

More on this saga... I think it was for the best that my old install got messed up. First, I discovered that I was wrong thinking that old computer was 32-bit. It was 64-bit, so I have installed CentOS7.8 (2003) 64 bit version on it. I thought if I could get it back up in a day I'd have time to get the other machine looked over, copy old stuff off it, etc.

I connected the external USB drive with NTFS on it to my windows PC. I could write fast to it for a while, then I tried copying a bunch of small files again (my music library) and EVEN ON WINDOWS it slowed to a crawl just like on Linux. I suspect it is an NTFS issue and had nothing to do with Linux in the first place - so I'm defragging it once more, which sped it up yesterday.

So anyway, it's been a roller-coaster. I don't know what advice at this point to ask for, I think I have the situation understood. Thanks for the responses!

Oh Trevor - the comms errors were being given to me by WinSCP. They only happened when the write speed was getting super slow. Don't know exactly what to say, they aren't happening now that the machine is on CentOS 7.8 64-bit, and they only ever happened when writing files to the USB drive on Linux, never the boot drive, so seemed directly related to that. At this point that OS is dead and I erased the disk so no opportunity for postmortem on that. Thanks for the suggestions!

danielrg76
Posts: 5
Joined: 2020/07/11 09:04:18

Re: abysmal NTFS write speeds USB drive

Post by danielrg76 » 2020/07/13 09:24:12

All right. My new PLEX server is up with the latest CentOS 7.8 64-bit. Much nicer, and faster too. Kind of glad for the turn of events.

So this is all OBE now. And turns out the drive is just very slow in some cases (confirmed on Linux AND Windows), and might be an SMB drive too (Shingled Magnetic Recording) making it very slow for some writes, particularly when fragmented and/or more than half full.

Post Reply

Return to “CentOS 7 - General Support”