Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: spf13/afero
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.3.5
Choose a base ref
...
head repository: spf13/afero
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.4.0
Choose a head ref
  • 4 commits
  • 8 files changed
  • 3 contributors

Commits on Sep 7, 2020

  1. Copy the full SHA
    cb8b6bc View commit details

Commits on Sep 13, 2020

  1. Removing use of testify

    LandonTClipp committed Sep 13, 2020

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    bf960e8 View commit details

Commits on Sep 14, 2020

  1. Merge pull request #264 from LandonTClipp/memmapfs_lstatifpossible

    Make MemMapFs implement Lstater
    0xmichalis authored Sep 14, 2020
    Copy the full SHA
    277f220 View commit details
  2. Add tarfs implementation (reopen #265) (#266)

    * Initial commit for tarfs
    
    * tarfs: reword "open" status field
    
    * tarfs: use TestMain for FS setup
    
    We want to have the FS variable available through all the tests, so we
    we use a common "setup" function to initialise it.
    
    * tarfs: test: early exit for nonexisting files
    
    * tarfs: create test for filesystem Open
    
    * tarfs: implement File.Stat
    
    * tarfs: implement Fs.Open
    
    * tarfs: return error on non-supported methods
    
    As tarfs is a read-only filesystem backend, we return EROFS (Read-only
    file system) from any method that makes modifications.
    
    * tarfs: implement File.data as bytes.Reader
    
    Most of the operations that we want to implement for tarfs.File are
    already defined in bytes.Reader.
    
    We could use a plain slice and implement all the seeking manually, but I
    think using this is more convenient.
    
    * tarfs: short format for simple methods
    
    * tarfs: add missing closing brace in tests
    
    * tarfs: add test for File.ReadAt
    
    * tarfs: test File.ReadAt
    
    * tarfs: add tests for File.Read
    
    * tarfs: implement File.Read
    
    * tarfs: add tests for File.Seek
    
    * tarfs: implement File.Seek
    
    * tarfs: add tests for File.Name
    
    * tarfs: implement File.Name
    
    * tarfs: add tests for File.Close
    
    * tarfs: implement File.Close
    
    * tarfs: add tests for OpenFile
    
    * tarfs: fix test for Fs.OpenFile
    
    If the call fails, we don't have to close the file
    
    * tarfs: remove code not needed after using filepath.Clean
    
    * tarfs: Open: return a copy of the internal structure
    
    As we modify the struct fields when closing, we don't want to lose the
    internal representation of the file, in case we want to reopen it.
    
    Return a copy of the File, although we keep using the same pointers to
    tar.Header and buffer.Reader. Maybe we will need to change that in the
    future.
    
    * tarfs: implement Fs.OpenFile
    
    * tarfs: use Fatalf for unexpected error in TestFsOpen
    
    * tarfs: add tests for Fs.Stat
    
    * tarfs: implement Fs.Stat
    
    * tarfs: remove TestNewFs
    
    That test depends too much on the internal imlementation, and it is
    easier to break if we change it.
    
    * tarfs: remove unused code
    
    * tarfs: change internal implementation
    
    To be able to handle directories (File.Readdir, File.Readdirnames), the
    naive single-map implementation makes it a bit harder to implement.
    
    Inspired by the zipfs backend, switch to an internal implementation of a
    map of directories that contains a map of files, so the directory
    methods are easier to implement.
    
    Also, treat the "virtual" filesystem as absolute, just like zipfs does.
    
    * tarfs: use Fatal errors to avoid panics
    
    * tarfs: add pseudoroot
    
    * tarfs: add tests for File.Readdir
    
    * tarfs: add pointer Fs in the File structure
    
    For directory-related operations we will need to access the internal
    structure in the Fs.
    
    As Readdir and Readdirnames are File methods, we need to access such
    structure from the File.
    
    * tarfs: fix error
    
    * tarfs: use just the names for TestReaddir, easier than using fill os.FileInfo entries
    
    * tarfs: create a copy of the original entry when opening a file
    
    We added the fs field in the File struct to reference the underlying Fs
    object, but in the Open cal we were not passing it, making all the
    opened files to have a nil pointer in that field.
    
    Change to make a copy of the original file, and returning that
    
    * tarfs: implement File.Readdir
    
    * tarfs: add tests for File.Readdirnames
    
    * tarfs: implement Readdirnames
    
    * tarfs: add test for File.Name
    
    * tarfs: change tests to use the Afero interface instead
    
    * tarfs: add tests for Glob from zipfs
    
    * tarfs: update main repo references to tarfs
    
    * tarfs: use OS-specific file separator for pseudoroot
    
    * tarfs: fix path handling in Windows systems
    agimenez authored Sep 14, 2020
    Copy the full SHA
    a4ea980 View commit details
Showing with 714 additions and 2 deletions.
  1. +1 −1 .travis.yml
  2. +0 −1 README.md
  3. +5 −0 memmap.go
  4. +28 −0 memmap_test.go
  5. +144 −0 tarfs/file.go
  6. +134 −0 tarfs/fs.go
  7. +402 −0 tarfs/tarfs_test.go
  8. BIN tarfs/testdata/t.tar
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
@@ -19,4 +19,4 @@ script:
- go build -v ./...
- go test -count=1 -cover -race -v ./...
- go vet ./...
- FILES=$(gofmt -s -l . zipfs sftpfs mem); if [[ -n "${FILES}" ]]; then echo "You have go format errors; gofmt your changes"; exit 1; fi
- FILES=$(gofmt -s -l . zipfs sftpfs mem tarfs); if [[ -n "${FILES}" ]]; then echo "You have go format errors; gofmt your changes"; exit 1; fi
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -380,7 +380,6 @@ The following is a short list of possible backends we hope someone will
implement:

* SSH
* TAR
* S3

# About the project
5 changes: 5 additions & 0 deletions memmap.go
Original file line number Diff line number Diff line change
@@ -317,6 +317,11 @@ func (m *MemMapFs) Rename(oldname, newname string) error {
return nil
}

func (m *MemMapFs) LstatIfPossible(name string) (os.FileInfo, bool, error) {
fileInfo, err := m.Stat(name)
return fileInfo, false, err
}

func (m *MemMapFs) Stat(name string) (os.FileInfo, error) {
f, err := m.Open(name)
if err != nil {
28 changes: 28 additions & 0 deletions memmap_test.go
Original file line number Diff line number Diff line change
@@ -650,3 +650,31 @@ func TestMemFsOpenFileModeIllegal(t *testing.T) {
t.Fatalf("should not be able to use OpenFile to set illegal mode: %s", info.Mode().String())
}
}

// LstatIfPossible should always return false, since MemMapFs does not
// support symlinks.
func TestMemFsLstatIfPossible(t *testing.T) {
t.Parallel()

fs := NewMemMapFs()

// We assert that fs implements Lstater
fsAsserted, ok := fs.(Lstater)
if !ok {
t.Fatalf("The filesytem does not implement Lstater")
}

file, err := fs.OpenFile("/a.txt", os.O_CREATE, 0o644)
if err != nil {
t.Fatalf("Error when opening file: %v", err)
}
defer file.Close()

_, lstatCalled, err := fsAsserted.LstatIfPossible("/a.txt")
if err != nil {
t.Fatalf("Function returned err: %v", err)
}
if lstatCalled {
t.Fatalf("Function indicated lstat was called. This should never be true.")
}
}
144 changes: 144 additions & 0 deletions tarfs/file.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
package tarfs

import (
"archive/tar"
"bytes"
"os"
"path/filepath"
"sort"
"syscall"

"github.com/spf13/afero"
)

type File struct {
h *tar.Header
data *bytes.Reader
closed bool
fs *Fs
}

func (f *File) Close() error {
if f.closed {
return afero.ErrFileClosed
}

f.closed = true
f.h = nil
f.data = nil
f.fs = nil

return nil
}

func (f *File) Read(p []byte) (n int, err error) {
if f.closed {
return 0, afero.ErrFileClosed
}

if f.h.Typeflag == tar.TypeDir {
return 0, syscall.EISDIR
}

return f.data.Read(p)
}

func (f *File) ReadAt(p []byte, off int64) (n int, err error) {
if f.closed {
return 0, afero.ErrFileClosed
}

if f.h.Typeflag == tar.TypeDir {
return 0, syscall.EISDIR
}

return f.data.ReadAt(p, off)
}

func (f *File) Seek(offset int64, whence int) (int64, error) {
if f.closed {
return 0, afero.ErrFileClosed
}

if f.h.Typeflag == tar.TypeDir {
return 0, syscall.EISDIR
}

return f.data.Seek(offset, whence)
}

func (f *File) Write(p []byte) (n int, err error) { return 0, syscall.EROFS }

func (f *File) WriteAt(p []byte, off int64) (n int, err error) { return 0, syscall.EROFS }

func (f *File) Name() string {
return filepath.Join(splitpath(f.h.Name))
}

func (f *File) getDirectoryNames() ([]string, error) {
d, ok := f.fs.files[f.Name()]
if !ok {
return nil, &os.PathError{Op: "readdir", Path: f.Name(), Err: syscall.ENOENT}
}

var names []string
for n := range d {
names = append(names, n)
}
sort.Strings(names)

return names, nil
}

func (f *File) Readdir(count int) ([]os.FileInfo, error) {
if f.closed {
return nil, afero.ErrFileClosed
}

if !f.h.FileInfo().IsDir() {
return nil, syscall.ENOTDIR
}

names, err := f.getDirectoryNames()
if err != nil {
return nil, err
}

d := f.fs.files[f.Name()]
var fi []os.FileInfo
for _, n := range names {
if n == "" {
continue
}

f := d[n]
fi = append(fi, f.h.FileInfo())
if count > 0 && len(fi) >= count {
break
}
}

return fi, nil
}

func (f *File) Readdirnames(n int) ([]string, error) {
fi, err := f.Readdir(n)
if err != nil {
return nil, err
}

var names []string
for _, f := range fi {
names = append(names, f.Name())
}

return names, nil
}

func (f *File) Stat() (os.FileInfo, error) { return f.h.FileInfo(), nil }

func (f *File) Sync() error { return nil }

func (f *File) Truncate(size int64) error { return syscall.EROFS }

func (f *File) WriteString(s string) (ret int, err error) { return 0, syscall.EROFS }
134 changes: 134 additions & 0 deletions tarfs/fs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
// package tarfs implements a read-only in-memory representation of a tar archive
package tarfs

import (
"archive/tar"
"bytes"
"io"
"os"
"path/filepath"
"syscall"
"time"

"github.com/spf13/afero"
)

type Fs struct {
files map[string]map[string]*File
}

func splitpath(name string) (dir, file string) {
name = filepath.ToSlash(name)
if len(name) == 0 || name[0] != '/' {
name = "/" + name
}
name = filepath.Clean(name)
dir, file = filepath.Split(name)
dir = filepath.Clean(dir)
return
}

func New(t *tar.Reader) *Fs {
fs := &Fs{files: make(map[string]map[string]*File)}
for {
hdr, err := t.Next()
if err == io.EOF {
break
}
if err != nil {
return nil
}

d, f := splitpath(hdr.Name)
if _, ok := fs.files[d]; !ok {
fs.files[d] = make(map[string]*File)
}

var buf bytes.Buffer
size, err := buf.ReadFrom(t)
if err != nil {
panic("tarfs: reading from tar:" + err.Error())
}

if size != hdr.Size {
panic("tarfs: size mismatch")
}

file := &File{
h: hdr,
data: bytes.NewReader(buf.Bytes()),
fs: fs,
}
fs.files[d][f] = file

}

// Add a pseudoroot
fs.files[afero.FilePathSeparator][""] = &File{
h: &tar.Header{
Name: afero.FilePathSeparator,
Typeflag: tar.TypeDir,
Size: 0,
},
data: bytes.NewReader(nil),
fs: fs,
}

return fs
}

func (fs *Fs) Open(name string) (afero.File, error) {
d, f := splitpath(name)
if _, ok := fs.files[d]; !ok {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
}

file, ok := fs.files[d][f]
if !ok {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
}

nf := *file

return &nf, nil
}

func (fs *Fs) Name() string { return "tarfs" }

func (fs *Fs) Create(name string) (afero.File, error) { return nil, syscall.EROFS }

func (fs *Fs) Mkdir(name string, perm os.FileMode) error { return syscall.EROFS }

func (fs *Fs) MkdirAll(path string, perm os.FileMode) error { return syscall.EROFS }

func (fs *Fs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error) {
if flag != os.O_RDONLY {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.EPERM}
}

return fs.Open(name)
}

func (fs *Fs) Remove(name string) error { return syscall.EROFS }

func (fs *Fs) RemoveAll(path string) error { return syscall.EROFS }

func (fs *Fs) Rename(oldname string, newname string) error { return syscall.EROFS }

func (fs *Fs) Stat(name string) (os.FileInfo, error) {
d, f := splitpath(name)
if _, ok := fs.files[d]; !ok {
return nil, &os.PathError{Op: "stat", Path: name, Err: syscall.ENOENT}
}

file, ok := fs.files[d][f]
if !ok {
return nil, &os.PathError{Op: "stat", Path: name, Err: syscall.ENOENT}
}

return file.h.FileInfo(), nil
}

func (fs *Fs) Chmod(name string, mode os.FileMode) error { return syscall.EROFS }

func (fs *Fs) Chtimes(name string, atime time.Time, mtime time.Time) error { return syscall.EROFS }
Loading