* lib/fts.c [!_LIBC]: Do not include dirent--.h, unistd--.h, cloexec.h.
(opendirat, diropen): Use O_CLOEXEC instead of set_cloexec_flag.
(fts_build): Use F_DUPD_CLOEXEC rinstad of set_cloexec_flag.
(fd_ring_check): Set cloexec flag on new file descriptors.
(fts_build, fd_ring_check): While we’re at it, make sure the
resulting file descriptor is not 0, 1, or 2, since that is easy.
* modules/fts (Depends-on): Remove cloexec, dirent-safer, dup,
fcntl-safer, unistd-safer. Add fcntl.
* doc/posix-headers/stdalign.texi (stdalign.h):
Document the C11 restriction.
* lib/fts.c: Include stddef.h, for max_align_t.
(fts_alloc): Align using max_align_t, not FTSENT.
* modules/fts (Depends-on): Add stddef.
GCC 5.1.1 with -O2 and -fsanitize=address reports
the following from `chmod a+rx ..` for example:
ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61200000be48
READ of size 4 at 0x61200000be48 thread T0
#0 0x40f0a1 in fts_stat lib/fts.c:1821
#1 0x4141ec in fts_open lib/fts.c:514
#2 0x40ca6e in xfts_open lib/xfts.c:36
#3 0x402c35 in process_files src/chmod.c:336
#4 0x402c35 in main src/chmod.c:566
#5 0x7f8fdc38678f in __libc_start_main (/lib64/libc.so.6+0x2078f)
#6 0x404608 in _start (/home/padraig/git/coreutils/src/chmod+0x404608)
0x61200000be4b is located 0 bytes to the right of
267-byte region [0x61200000bd40,0x61200000be4b) allocated by thread T0 here:
#0 0x7f8fdd4cfa0a in malloc (/lib64/libasan.so.2+0x98a0a)
#1 0x40f22c in fts_alloc lib/fts.c:1916
The read of size 4 from a heap object of size 3 is indeed invalid,
though this may be due to incorrect padding assumptions by GCC, see:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66661
The read is coming from the a[2] access in ISDOT() which GCC seems
to assume can access according to the alignment of the main structure,
since the flexible array members are being accessed directly.
One could cast the parameter to ISDOT() to (char const*) to restrict
to byte at a time access, however it seems more correct and maintainable
to increase the buffer size to the appropriate alignment to avoid this issue.
For reference some notes on alignment and flexible array members are at:
https://sites.google.com/site/embeddedmonologue/home/c-programming/data-alignment-structure-padding-and-flexible-array-member
* lib/fts.c (fts_alloc): Increase allocation to alignof(FTSENT).
* modules/fts: Depend on stdalign.
Without this fix, GCC 5.1 (correctly) warns about a subscript
error on the fts_name component of FTSENT. It's actually a
flexible member, so define it that way on C99 or later hosts.
* lib/fts.c (fts_alloc): Use offsetof, not sizeof, for a
structure that now has a flexible array member.
* lib/fts_.h (__FLEXIBLE_ARRAY_MEMBER): New macro.
(FTSENT): fts_name is now flexible on C99-or-later platforms.
* modules/fts (Depends-on): Add flexmember.
* modules/fts (Depends-on): Depend on fdopendir. This is needed
on Solaris 8, at least, since it lacks fdopendir. Evidently the
problem was introduced when fdopendir was split out.
* lib/unistd.in.h (dup): Declare only if the 'dup' module is in use.
Allow replacement.
* lib/dup.c: New file.
* lib/fchdir.c (rpl_dup): Remove function.
* m4/dup.m4: New file.
* m4/fchdir.m4 (gl_FUNC_FCHDIR): Don't set REPLACE_DUP here.
* m4/unistd_h.m4 (gl_UNISTD_H): Test whether 'dup' is declared.
(gl_UNISTD_H_DEFAULTS): Initialize GNULIB_DUP.
* modules/unistd (Makefile.am): Substitute GNULIB_DUP.
* modules/dup: New file.
* tests/test-unistd-c++.cc: Check the signature of 'dup' only if the
'dup' module is in use.
* modules/fdopendir (Depends-on): Add dup.
* modules/fdutimensat-tests (Depends-on): Likewise.
* modules/fts (Depends-on): Likewise.
* modules/futimens-tests (Depends-on): Likewise.
* modules/posix_spawnp-tests (Depends-on): Likewise.
* modules/unistd-safer-tests (Depends-on): Likewise.
* modules/utimens-tests (Depends-on): Likewise.
* doc/posix-functions/dup.texi: Mention the new module and the problem
on MSVC.
* lib/fts.c (opendir): Remove #define; no longer used.
(opendirat): New arg PDIR_FD. All callers changed.
(fts_build, _opendir2): Use new opendirat to avoid the need for
dirfd, or for checking whether dirfd returns a negative value.
Don't use opendir; always use openat followed by fdopendir.
* lib/getcwd.c (__getcwd): Don't reset fd; fdopendir no longer clobbers
it.
* lib/glob.c (link_exists_p): Add comment explaining why dirfd never
returns -1 here.
* modules/fts (Depends-on): Remove dirfd.
* modules/getcwd (Depends-on): Likewise.
This avoids a quadratic, seek-related performance penalty when
operating on a directory containing many entries (measurable at 10k;
3.5 hours at 2 million entries with a cold cache) on certain types
of file systems, including ext3 and ext4, but not tmpfs.
* lib/fts.c (DT_MUST_BE, NOT_AN_INODE_NUMBER, D_INO): Define.
(FTS_INODE_SORT_DIR_ENTRIES_THRESHOLD): Define if not defined.
(S_MAGIC_TMPFS, S_MAGIC_NFS): Define.
(fs_handles_readdir_ordered_dirents_efficiently): New function.
(dirent_inode_sort_may_be_useful, fts_compare_ino): Likewise.
(fts_build): Set the stat.st_ino member from D_INO.
If it is likely to be useful, sort dirent entries on inode number.
* m4/fts.m4 (gl_FUNC_FTS_CORE): Check for fstatfs, sys/vfs.h,
and the struct statfs.f_type member.
* modules/fts (Depends-on): Add d-ino.
file descriptors. This also averts a failure on systems with
native openat support when a traversed directory lacks "x" access.
* lib/fts_.h: Include "i-ring.h"
(struct FTS) [fts_fd_ring]: New member.
* lib/fts.c (RESTORE_INITIAL_CWD): Also call fd_ring_clear.
(FCHDIR): Add parentheses.
(fd_ring_check, fd_ring_print) [!FTS_DEBUG]: Define away.
(cwd_advance_fd): Add a 3rd parameter. Adjust all callers.
When descending, rather than simply closing the previous
fts_cwd_fd value, push that file descriptor onto the ring.
(same_fd, fd_ring_print, fd_ring_check) [FTS_DEBUG]: New functions.
(fts_open): Initialize the new fd_ring member.
(fts_close): Clear the ring.
(fts_safe_changedir): When possible, use our new fd_ring to skip
the diropen and fstat and dev/ino comparison that would normally
accompany a virtual `chdir ("..")'.
* modules/fts (Depends-on): Add i-ring.
* modules/i-ring: New module.
* lib/i-ring.c, lib/i-ring.h, lib/i-ring-test.c: New files.
* m4/i-ring.m4: New file.
Avoid spurious inode-mismatch problems on non-POSIX file systems.
Details: http://article.gmane.org/gmane.comp.lib.gnulib.bugs/7416
* lib/fts_.h (FTS_DEFER_STAT): Define new flag.
(FTS_OPTIONMASK): Extend the mask to reflect this addition.
* lib/fts.c (DT_IS_KNOWN, DT_MUST_BE): Define.
(FTS_NO_STAT_REQUIRED, FTS_STAT_REQUIRED): Define.
(fts_set_stat_required): New function.
(fts_open): Defer the calls to fts_stat, if possible or requested.
Move the code that maps a command-line fts_info value FTS_DOT to FTS_D
into fts_stat itself.
(fts_read): Perform any required (deferred) fts_stat call.
(fts_build): Likewise, for the directory we're about to open and read.
In the readdir loop, carefully decide whether each entry will require
an eventual call to fts_stat, using dirent.d_type info if available.
(fts_stat): Move the test for whether to honor FTS_COMFOLLOW on
a command line argument into this function. Update all callers.
Map a return value of FTS_DOT to FTS_D for a command line argument.
* modules/fts (Depends-on): Add d-type. Alphabetize.
Thanks to Miklos Szeredi for his tenacity and for the initial
bug report about "find" failing on a FUSE-based file system.
* lib/__fpending.h: Add copyright notice.
* lib/fprintftime.h: Likewise.
* lib/savedir.c: Use (C) in copyright notice.
* lib/savedir.h: Likewise.
2006-08-15 Jim Meyering <jim@meyering.net>
* lib/at-func.c: New file, with the logic of all emulated at-functions.
* lib/openat-priv.h: Include <errno.h> and define ENOSYS,
in support of the EXPECTED_ERRNO macro.
* lib/openat.c (fstatat, unlinkat, fchownat): Remove function definitions.
Instead, define the appropriate symbols and include "at-func.c".
* lib/mkdirat.c (mkdirat): Likewise.
* lib/fchmodat.c (fchmodat): Likewise.
(ENOSYS): Remove definition.
* lib/openat.c: Don't include <errno.h>, now that "openat-priv.h" does it.
Don't include "unistd--.h" -- it wasn't ever used.
2006-01-17 Jim Meyering <jim@meyering.net>
Rewrite fts.c not to change the current working directory,
by using openat, fstatat, fdopendir, etc..
* lib/fts.c [! _LIBC]: Include "openat.h" and "unistd--.h".
(HAVE_OPENAT_SUPPORT): Define.
[_LIBC] (fchdir): Don't undef or define; no longer used.
(FCHDIR): Define in terms of cwd_advance_fd rather than fchdir.
Now, this `function' always succeeds, and consumes its file descriptor
parameter -- so callers must not close such FDs. Update callers.
(diropen_fd, opendirat, cwd_advance_fd): New functions.
(diropen): Add parameter, SP. Adjust all callers.
Implement using diropen_fd, rather than open.
(fts_open): Initialize new member, fts_cwd_fd.
Remove fts_rft-setting code.
(fts_close): Close fts_cwd_fd, if necessary.
(__opendir2): Define in terms of opendir or opendirat,
depending on whether the FST_NOCHDIR flag is set.
(fts_build): Since fts_safe_changedir consumes its FD, and since
this code must do `closedir(dirp)', dup the dirfd(dirp) argument,
and close the dup'd file descriptor upon failure.
(fts_stat): Use fstatat(...AT_SYMLINK_NOFOLLOW) in place of lstat.
(fts_safe_changedir): Tweak semantics to reflect that this function
now calls cwd_advance_fd and hence consumes its FD argument.
* lib/fts_.h [struct FTS] (fts_cwd_fd): New member.
[struct FTS] (fts_rft): Remove now-unused member.
[struct FTS] (fts_cycle.state): Improve comment.
* lib/openat.c (openat_needs_fchdir): New function.
* lib/openat.h (openat_needs_fchdir): Declare it.
2006-08-15 Jim Meyering <jim@meyering.net>
* m4/openat.m4 (gl_FUNC_OPENAT): Add at-func.c via AC_LIBSOURCES.
2006-01-17 Jim Meyering <jim@meyering.net>
* m4/fts.m4 (gl_FUNC_FTS_CORE): Depend on gl_FUNC_OPENAT.
2006-01-11 Jim Meyering <jim@meyering.net>
* m4/openat.m4 (gl_FUNC_OPENAT): Require and compile fchmodat.c.
Check for the lchmod function.
* modules/fts-lgpl (Depends-on): Remove gettext.
* lib/fts.c: Don't worry about debugging on pre-C99-comopatible hosts;
the configuration hassle isn't worth it.
Include inttypes.h and stdint.h unconditionally if FTS_DEBUG.
(LONGEST_MODIFIER, PRIuMAX): Remove.
* m4/fts.m4 (gl_FUNC_FTS_CORE): Don't check for inttypes.h or stdint.h,
and don't require gt_INTTYPES_PRI.
* MODULES.html.sh (File system functions): Add fts, fts-lgpl.
* lib/fts.c: Don't include "cycle-check.h" or "hash.h".
(setup_dir, free_dir): New functions.
(enter_dir, leave_dir): Define trivial
alternatives of _LGPL_PACKAGE. Move to fts-cycle.c if !_LGPL_PACKAGE.
(HT_INITIAL_SIZE, ENTER_DIR): Remove. All uses removed.
(LEAVE_DIR): Fix typo: pass Fts and Ent to leave_dir.
(struct Active_dir, AD_compare, AD_hash, enter_dir, leave_dir):
Move to fts-cycle.c.
(fts_open): Use setup_dir.
(fts_close): Use free_dir.
(fts_read): Have just one copy of the ENTER_DIR code rather than three.
This adds a label and some gotos, but the alternatives were messier.
Check for memory allocation failure when entering a dir.
(fts_stat) [_LGPL_PACKAGE]: Bring back glibc cycle detection code.
* lib/fts_.h (_LGPL_PACKAGE) [defined _LIBC]: New macro.
(FTS): New member fts_cycle, that is a union that contains the
old active_dir_ht and cycle_state. All uses changed to mention
fts_cycle.ht and fts_cycle.state.
* lib/fts-cycle.c: New file, containing GPL'ed code migrated out of
fts.c, with the following changes:
(setup_dir, free_dir): New functions.
(enter_dir): Now returns bool. Return true if successful, false
if memory exhausted. All callers changed.
Do not bother partly cleaning up on
memory allocation failure; that is free_dir's job.
However, free ad if hash_insert fails, to avoid memory leak.
(enter_dir, leave_dir): Accommodate change to FTS by inspecting
fts->fts_options to see which union member to use.
* m4/fts.m4 (gl_FUNC_FTS_CORE): Renamed from gl_FUNC_FTS.
(gl_FUNC_FTS, gl_FUNC_FTS_LGPL): New macros.
* lib/unlinkdir.h (cannot_unlink_dir) [UNLINK_CANNOT_UNLINK_DIR]:
Now a macro, to pacify GCC.
Paul Eggert