Copying files on Centos

General support questions
Post Reply
nanakwame
Posts: 3
Joined: 2020/07/21 05:52:15

Copying files on Centos

Post by nanakwame » 2020/07/21 06:11:15

So Here is the issue.
I have two 4 directories under this directory.
var/www/html/directory/directory/directory/{directory1, directory 2, directory 3, directory 4}
The total size of the four directories is approximately 95G.
Each directory contains files more than 40,000. I am supposed to copy the directory containing the files to an external hard drive
I connected an external hard drive with fat32 filesystem(MS-DOS) and 500GB.
It was mounted at /run/media/root/VolumeName.

I copied the first 2 directories and ran into an error saying space is full on the /run/media/root/VolumeName even though there is space on the hard drive. So I ejected the drive and moved the two directories I have copied to my machine. I came back to copy the remaining tow directories And had the same error that space is full.

It is noteworthy that when I started copying directory 3, I get an error space full when it copies to a point. But regardless I can copy directory 4 up to a certain point as well and then memory full will pop up.

Additional Information.
df -h shows there is space(below)
Filesystem Size Used Avail Use% Mounted on
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 63G 60M 63G 1% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/mapper/centos-root 2.5T 7.2G 2.3T 1% /
/dev/sda2 56G 221M 56G 1% /boot
/dev/sda1 60G 12M 60G 1% /boot/efi
/dev/mapper/centos-home 99G 66M 94G 1% /home
/dev/mapper/centos-var 265G 101G 165G 38% /var
tmpfs 13G 8.0K 13G 1% /run/user/42
tmpfs 13G 36K 13G 1% /run/user/0
tmpfs 13G 60K 13G 1% /run/user/1000
/dev/sdc1 466G 5.3G 461G 2% /ixi
// If youll notice the external is /dev/sdc1 and ive changed the mount to /ixi. Still error persists
// df -i doesnt show inode usage for /dev/sdc1

du -sh shows this (below)0 bin
200M boot
0 dev
64M etc
5.8M home
4.0K ixi
0 lib
0 lib64
16K lost+found
4.0K media
8.0K mnt
8.0K opt
du: cannot access 'proc/20137': No such file or directory
du: cannot access 'proc/20138': No such file or directory
du: cannot access 'proc/20146/task/20146/fd/3': No such file or directory
du: cannot access 'proc/20146/task/20146/fdinfo/3': No such file or directory
du: cannot access 'proc/20146/fd/3': No such file or directory
du: cannot access 'proc/20146/fdinfo/3': No such file or directory
0 proc
1.3G root
du: cannot access 'run/user/1000/gvfs': Permission denied
5.4G run
0 sbin
4.0K srv
0 sys
712K tmp
5.8G usr
101G var

Any help will be apprecited.

User avatar
jlehtone
Posts: 2921
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Copying files on Centos

Post by jlehtone » 2020/07/21 08:49:59

nanakwame wrote:
2020/07/21 06:11:15
I connected an external hard drive with fat32 filesystem(MS-DOS)
Every filesystem has different limits.

The question is: what limits does that filesystem have?
Not just size, but number of files (total or per directory) too.
Furthermore, what is the maximum size for a single file?

Could you use rsync to copy over network?

desertcat
Posts: 761
Joined: 2014/08/07 02:17:29
Location: Tucson, AZ

Re: Copying files on Centos

Post by desertcat » 2020/07/22 09:41:35

nanakwame wrote:
2020/07/21 06:11:15
So Here is the issue.
I have two 4 directories under this directory.
var/www/html/directory/directory/directory/{directory1, directory 2, directory 3, directory 4}
The total size of the four directories is approximately 95G.
Each directory contains files more than 40,000. I am supposed to copy the directory containing the files to an external hard drive
I connected an external hard drive with fat32 filesystem(MS-DOS) and 500GB.
It was mounted at /run/media/root/VolumeName.

I copied the first 2 directories and ran into an error saying space is full on the /run/media/root/VolumeName even though there is space on the hard drive. So I ejected the drive and moved the two directories I have copied to my machine. I came back to copy the remaining tow directories And had the same error that space is full.

It is noteworthy that when I started copying directory 3, I get an error space full when it copies to a point. But regardless I can copy directory 4 up to a certain point as well and then memory full will pop up.

Additional Information.
df -h shows there is space(below)
Filesystem Size Used Avail Use% Mounted on
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 63G 60M 63G 1% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/mapper/centos-root 2.5T 7.2G 2.3T 1% /
/dev/sda2 56G 221M 56G 1% /boot
/dev/sda1 60G 12M 60G 1% /boot/efi
/dev/mapper/centos-home 99G 66M 94G 1% /home
/dev/mapper/centos-var 265G 101G 165G 38% /var
tmpfs 13G 8.0K 13G 1% /run/user/42
tmpfs 13G 36K 13G 1% /run/user/0
tmpfs 13G 60K 13G 1% /run/user/1000
/dev/sdc1 466G 5.3G 461G 2% /ixi
// If youll notice the external is /dev/sdc1 and ive changed the mount to /ixi. Still error persists
// df -i doesnt show inode usage for /dev/sdc1

du -sh shows this (below)0 bin
200M boot
0 dev
64M etc
5.8M home
4.0K ixi
0 lib
0 lib64
16K lost+found
4.0K media
8.0K mnt
8.0K opt
du: cannot access 'proc/20137': No such file or directory
du: cannot access 'proc/20138': No such file or directory
du: cannot access 'proc/20146/task/20146/fd/3': No such file or directory
du: cannot access 'proc/20146/task/20146/fdinfo/3': No such file or directory
du: cannot access 'proc/20146/fd/3': No such file or directory
du: cannot access 'proc/20146/fdinfo/3': No such file or directory
0 proc
1.3G root
du: cannot access 'run/user/1000/gvfs': Permission denied
5.4G run
0 sbin
4.0K srv
0 sys
712K tmp
5.8G usr
101G var

Any help will be apprecited.
OK my favorite utility of all utility programs dates back to M$-DOS, and was included in a suite of utility programs called "Norton Utilities". In that group was something called Norton Commander (nc). It was a File Browser and many other things as well. Then about 20 years ago I kissed M$ goodby and jumped to Linux I discovered an updated and more powerful version of nc, called Midnight Commander (mc). If you have not got mc downloaded may I suggest that you do so?!? 95% of all files. dir., etc., can be edited or copied with ease. You have a 2 panel design. In one Panel you browse over to the the directory where the sub-directory is contained. In the 2 pane you have the location you want to copy the sub-directory to (this would be your external drive). Now hit <= Tab => and it will take you back to the originating. Highlight the directory you want to copy hit F5. It will tell you if that it will copy Sub-directory to the -- in your case -- external drive. At this point you can jazz things up by creating a Super-Directory -- a Directory that contains all the other sub-directories etc. The sky's the limit. IF there are are internal problems in some of the directories you should be able to see them by scrolling down the contents of the files in the sub-directory. Very rarely I will encounter some problem that can't be solved my mc, and in that case I bring out the BIG GUN: Sublime or resort to CLI commands, but by using mc I already have a feeling what I will find. Unless there are pointers that link one sub-direcrory files to other dependent sub-sub-directory files with certain builtin space limitations in which case just simply copy the Master Directory to the the external drive. The only limit you should then encounter is the size of the external drive.

Regardless mc is one of my "Can't-Live-Without" programs. For me it is a time saver over having to use the straight CLI which I use only if necessary.

pjsr2
Posts: 483
Joined: 2014/03/27 20:11:07

Re: Copying files on Centos

Post by pjsr2 » 2020/07/22 12:10:44

The problem is not the space on the disk, but the space in a directory on the disk to accommodate an additional file.

The maximum size for a FAT32 directory is 65536 32-bit entries. A file will require at least 2 entries depending on the length of the filename. So directories with > 40000 files will not fly. Reformat the hard disk with an ext4 filesystem.
File size on FAT32 is limited to 4 GiB, so storing the files as a tar-archive is not an option.

Note that when you copy files to a filesystem like fat32 or ntfs, you lose permission and ownership information on the files. You also lose SElinux contexts. Fat32 has a considerably lower resolution for time stamps, so you lose on that as well.

nanakwame
Posts: 3
Joined: 2020/07/21 05:52:15

Re: Copying files on Centos

Post by nanakwame » 2020/07/23 13:19:37

Ohkay. Ill try this. But if I am done copying, will my PC recognise the ext4 filesystem on the external drive if I connect so I get the files to the appropriate quarters

pjsr2
Posts: 483
Joined: 2014/03/27 20:11:07

Re: Copying files on Centos

Post by pjsr2 » 2020/07/23 15:21:08

If the other PC uses Linux, it will certainly work.

However, when that PC has an operating system from Microsoft, it will not be able to read an ext4 partition out of the box. Microsoft is not very good in interoperability. You can find some solutions on https://helpdeskgeek.com/how-to/how-to- ... n-windows/. Anyhow, when you copy the files to a Windows machine you have to be very careful. File names may contain characters that are not allowed in Windows file names. Files with names that only differ in upper/lower case may silently overwrite each other.

nanakwame
Posts: 3
Joined: 2020/07/21 05:52:15

Re: Copying files on Centos

Post by nanakwame » 2020/07/27 14:34:19

Thanks very much. I copied the files over the network. I formatted the drive and used ext4 all the files were able to copy so the problem was definitely the file system fat 32. But I later copied over the network remotely from my pc.

To benefit any future individual who might stumble upon this Ill explain the procedure.
Assuming my PC is A and the Server is B. You can use secure copy over ssh.

from my PC A

scp -r user@PCB:/path/to/folder/containing/files /location/of/PC/A/where/you/want/the/files

the -r will repetitively copy all the files.

Post Reply

Return to “CentOS 7 - General Support”