Version 3 (modified by boto, 5 years ago)

--

Loading from a virtual file system

Overview

In this tutorial you will see how to implement a mechanism for reading osg resources -- such as images, osg and ive files, fonts, etc. -- from a virtual file system. You may ask "what is it good for?". In short: using a virtual file system allows having a couple of archive files e.g. in zip or other compressed formats containing all necessary data for your application, instead of having the entire data folder hierarchy and files in your application installation folder. I won't dive into a discussion about the motivation for using a virtual file system. I just explain how to manage osg file readers fetching the actual data from a virtual file system. "What does this mean?", you may ask. Here follows a pseudo code snippet showing what I mean.

// trying to load an osg file using the traditional mechanism
osg::ref_ptr< osg::Node > node = osgDB::readNodeFile( "/path to my file/my file.osg" );
...

What here happens is that in osg db the proper file reader plugin for reading files with extension osg is searched and used for loading the file directly from mass storage, e.g. hard disk.

Now consider this:

// setup the virtual file system

Virtual File System Manager -> initialize();
Virtual File System Manager -> mount( "my archive file name, e.g. a zip file", "/" ); // this mounts the archive into root path "/"

VFSStream vfsstream = Virtual File System Manager -> getStream( "/mesh/cow.osg" ); // this file is actually in the zip file archive

if ( vfsstream.valid() )
{
	osg::ref_ptr< osg::Node > node = osgDB::readNodeFile( vfsstream );
	...
}

In this case the file /mesh/cow.osg is read from the virtual file system which mounted your archive file to root path "/".

In the example above the osg file itself could be loaded using provided special stream object vfsstream. However, if the osg file refers to external resources such as image files then they would not get loaded via the given stream. The default behaviour is opening the referenced image file using its file name, which in consequence causes the image loader plugin to create an own stream by opening the file from mass storage. In order to avoid this we will implement a new loader plugin and register it in osg db. This plugin will be able to hook into osg db's task for searching a proper loader at right time and feed our special stream object to such a loader.

The code for using the new plugin looks like this.

// setup the virtual file system

Virtual File System Manager -> initialize();
Virtual File System Manager -> mount( "my archive file name, e.g. a zip file", "/" ); // this mounts the archive into root path "/"
...

osg::ref_ptr< osg::Node > node = osgDB::readNodeFile( "/mesh/cow.osg.physfs" ); // the file /mesh/cow.osg is actually in the zip archive
...

As you may have noticed, the osg file ends with physfs suffix this time. Our new plugin will accept file types with suffix physfs, it is.

We are going to implement:

  • a stream class which is derived from std::streambuf class and reads the data from a virtual file system. The virtual file system functionality itself is provided by PhysFS library (see http://icculus.org/physfs)
  • a new osg plugin which accepts the suffix ".physfs".

The code uses the PhysFS library and some utilities from Yaf3D framework (see Conclusion below).

Code: Special std::streambuf class

This class implements our special stream buffer. Do not bother about RefCount? class, the macro DECLARE_SMARTPTR_ACCESS and other foreign classes and functions. The concept behind the code is independent from those utilities.

//! A class containing the content of a file as std::streambuf
/** Use following code for making an std::istream out of it.

    StreambufPtr ibuf( "a vfs path" );
    if ( !ibuf->ok() )
        return file could not be found;

    std::istream input( ibuf.getRef() );
    ...
*/
class Streambuf : public std::streambuf, public RefCount< Streambuf >
{
    //! Set the smart pointer class as friend
    DECLARE_SMARTPTR_ACCESS( Streambuf )

    public:

        typedef std::streambuf::pos_type        pos_type;
        typedef std::streambuf::off_type        off_type;
        typedef std::ios::seekdir               seekdir;
        typedef std::ios::openmode              openmode;

        explicit                                Streambuf( const std::string& filename );

                                                ~Streambuf();

        bool                                    ok() const;

    protected:

        // Avoid copy and assignment
                                                Streambuf( const Streambuf& );

                                                Streambuf &operator = ( const Streambuf& );

        bool                                    readVFSFile( const std::string& filename );

        // Overridden methods
        int_type                                uflow();

        int_type                                underflow();

        int_type                                pbackfail( int_type ch );

        std::streamsize                         showmanyc();

        Streambuf::pos_type                     seekoff( Streambuf::off_type off, Streambuf::seekdir way, Streambuf::openmode = ( std::ios_base::in | std::ios_base::out ) );

        Streambuf::pos_type                     seekpos( Streambuf::pos_type pos, Streambuf::openmode = ( std::ios_base::in | std::ios_base::out ) );

        char*                                   _p_begin;

        char *                                  _p_end;

        char *                                  _p_current;

        bool                                    _ok;

        std::streamsize                         _buffSize;
};

//! Typedef for the streambuf smart pointer
typedef SmartPtr< Streambuf >   StreambufPtr;


//****************************************************//


//! Implementation of streambuf class
Streambuf::Streambuf( const std::string& filename ) :
 _p_begin( NULL ),
 _p_end( NULL ),
 _p_current( NULL ),
 _ok( false ),
 _buffSize( 0 )
{
    _ok = readVFSFile( filename );
}

Streambuf::~Streambuf()
{
    if ( _p_begin )
        delete[] _p_begin;
}

bool Streambuf::ok() const
{
    return _ok;
}

Streambuf::int_type Streambuf::underflow()
{
    if ( _p_current == _p_end )
        return traits_type::eof();

    return static_cast< Streambuf::int_type >( *_p_current ) & 0x000000ff;
}

Streambuf::int_type Streambuf::uflow()
{
    if ( _p_current == _p_end )
        return traits_type::eof();

    return static_cast< Streambuf::int_type >( *_p_current++ ) & 0x000000ff;
}

Streambuf::int_type Streambuf::pbackfail( int_type ch )
{
    if ( _p_current == _p_begin || (ch != traits_type::eof() && ch != _p_current[ -1 ] ) )
        return traits_type::eof();

    return static_cast< Streambuf::int_type >( *--_p_current ) & 0x000000ff;
}

std::streamsize Streambuf::showmanyc()
{
    return _p_end - _p_current;
}

Streambuf::pos_type Streambuf::seekoff( Streambuf::off_type off, Streambuf::seekdir way, Streambuf::openmode /*mode*/ )
{
    char* p_pos = NULL;
    if ( way == std::ios_base::beg )
        p_pos = _p_begin;
    else if ( way == std::ios_base::end )
        p_pos = _p_end;
    else if ( way == std::ios_base::cur )
        p_pos = _p_current;

    if ( !p_pos )
        return traits_type::eof();

    if ( ( ( p_pos + off ) > _p_end ) || ( ( p_pos + off ) < _p_begin ) )
        return traits_type::eof();

    _p_current = &p_pos[ off ];

    return ( &p_pos[ off ] - _p_begin );
}

Streambuf::pos_type Streambuf::seekpos( Streambuf::pos_type pos, Streambuf::openmode /*mode*/ )
{
    return seekoff( pos, std::ios_base::beg );
}

bool Streambuf::readVFSFile( const std::string& filename )
{
    PHYSFS_File* p_handle = PHYSFS_openRead( filename.c_str() );
    if ( !handle )
    {
        log_warning << "VFS: cannot open requested file: " << filename << std::endl;
        return false;
    }

    PHYSFS_sint64 filesize = PHYSFS_fileLength( p_handle );
    if ( filesize < 0 )
    {
        log_warning << "VFS: cannot determine file size: " << filename << std::endl;
        return false;
    }

    char* buffer = new char[ filesize + 1 ];
    PHYSFS_sint64 cnt = PHYSFS_read( p_handle, buffer, filesize, 1 );
    PHYSFS_close( p_handle );

    if ( cnt < 0 )
    {
        log_warning << "VFS: cannot read file: " << filename << std::endl;
        return false;
    }

    _p_begin = buffer;
    _p_end   = &buffer[ filesize ];
    _p_current = _p_begin;
    _buffSize  = filesize;

    return true;
}

Code: physfs loader plugin

This plugin is actually a proxy around existing osg plugins -- similar to curl plugin which loads the content of files from network. Some specials about this plugin:

  • it has no write methods, it can only read files
  • it cannot read using a given std::istream, it works only given a file name
  • a particular trick is implemented in order to get sub-resources (e.g. images) in a file using the same plugin, although they do not have the file suffix physfs (see member variable _acceptAll below).
//! OSG Reader for physfs
class ReaderPhysFS : public osgDB::ReaderWriter
{
    public:

                                                    ReaderPhysFS();

        virtual                                     ~ReaderPhysFS() {}

        virtual const char*                         className() const;

        virtual bool                                acceptsExtension( const std::string& extension ) const;

        virtual osgDB::ReaderWriter::ReadResult     readObject( const std::string& file, const osgDB::ReaderWriter::Options* options ) const;

        virtual osgDB::ReaderWriter::ReadResult     readImage( const std::string& file, const osgDB::ReaderWriter::Options* options ) const;

        virtual osgDB::ReaderWriter::ReadResult     readNode( const std::string& file, const osgDB::ReaderWriter::Options* options ) const;

        virtual osgDB::ReaderWriter::ReadResult     readObject( std::istream& fin, const osgDB::ReaderWriter::Options* options ) const;

        virtual osgDB::ReaderWriter::ReadResult     readImage( std::istream& fin, const osgDB::ReaderWriter::Options* options ) const;

        virtual osgDB::ReaderWriter::ReadResult     readNode( std::istream& fin, const osgDB::ReaderWriter::Options* options ) const;

    protected:

        mutable bool                                _acceptAll;
};


//***********************************************************//

//! Implementation of physfs plugin
ReaderPhysFS::ReaderPhysFS() :
 _acceptAll( false )
{
    supportsExtension( "physfs","PhysFS based virtual file system loader" );
}

const char* ReaderPhysFS::className() const
{
    return "PhysFS Reader";
}

bool ReaderPhysFS::acceptsExtension( const std::string& extension ) const
{
    // we need this mode in order to let this reader handle also images and other sub-resources
    if ( _acceptAll )
        return true;

    if ( osgDB::equalCaseInsensitive( extension, "physfs" ) )
        return true;

    return false;
}

osgDB::ReaderWriter::ReadResult ReaderPhysFS::readObject( const std::string& file, const osgDB::ReaderWriter::Options* options ) const
{
#ifdef _DEBUG
    if ( !PHYSFS_isInit() )
    {
        osg::notify( osg::FATAL ) << "Error: PhysFS is not initialized" << std::endl;
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
    }
#endif

    return readNode( file, options );
}

osgDB::ReaderWriter::ReadResult ReaderPhysFS::readImage( const std::string& file, const osgDB::ReaderWriter::Options* options ) const
{
#ifdef _DEBUG
    if ( !PHYSFS_isInit() )
    {
        osg::notify( osg::FATAL ) << "Error: PhysFS is not initialized" << std::endl;
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
    }
#endif

    // try to find the proper path in vfs
    std::string path;
    if ( options )
    {
        osgDB::FilePathList pl = options->getDatabasePathList();
        while ( pl.size() )
        {
            std::string searchpath = pl.front();
            pl.pop_back();
            PHYSFS_File* p_handle = PHYSFS_openRead( yaf3d::cleanPath( searchpath + "/" + file ).c_str() );
            if ( p_handle )
            {
                path = searchpath;
                PHYSFS_close( p_handle );
                break;
            }
        }
    }

    // get the file from virtual file system and setup a stream
    Streambuf ibuf( path + "/" + yaf3d::cleanPath( file ) );
    if ( !ibuf.ok() )
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;

    std::istream input( &ibuf );

    // set the reader extension restriction
    _acceptAll = false;

    std::string ext = osgDB::getFileExtension( file );
    osgDB::ReaderWriter* p_reader = osgDB::Registry::instance()->getReaderWriterForExtension( ext );
    if ( !p_reader )
    {
        osg::notify( osg::FATAL ) << "ReaderPhysFS::readImage cannot find image reader for " << file << std::endl;
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
    }

    // set the filepath into list of seach paths
    osg::ref_ptr< osgDB::ReaderWriter::Options > lopt = options ? static_cast< osgDB::ReaderWriter::Options* >( options->clone( osg::CopyOp::SHALLOW_COPY ) ) : new osgDB::ReaderWriter::Options;
    lopt->getDatabasePathList().push_front( osgDB::getFilePath( file ) );

    _acceptAll = false;

    return p_reader->readImage( input, lopt );
}

osgDB::ReaderWriter::ReadResult ReaderPhysFS::readNode( const std::string& file, const osgDB::ReaderWriter::Options* options ) const
{
#ifdef _DEBUG
    if ( !PHYSFS_isInit() )
    {
        osg::notify( osg::FATAL ) << "Error: PhysFS is not initialized" << std::endl;
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
    }
#endif

    std::string vfsext = osgDB::getLowerCaseFileExtension( file );
    if ( !acceptsExtension( vfsext ) )
        return osgDB::ReaderWriter::ReadResult::FILE_NOT_HANDLED;

    std::string actualfilename = osgDB::getNameLessExtension( file );

    // get the file from virtual file system and setup a stream
    Streambuf ibuf( actualfilename );
    if ( !ibuf.ok() )
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;

    std::istream input( &ibuf );

    // set the reader extension restriction
    _acceptAll = false;

    std::string ext = osgDB::getFileExtension( actualfilename );
    osgDB::ReaderWriter* p_reader = osgDB::Registry::instance()->getReaderWriterForExtension( ext );
    if ( !p_reader )
    {
        osg::notify( osg::WARN ) << "ReaderPhysFS::readNode cannot find reader for " << actualfilename << std::endl;
        return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
    }

    // set the filepath into list of seach paths
    osg::ref_ptr< osgDB::ReaderWriter::Options > lopt = options ? static_cast< osgDB::ReaderWriter::Options* >( options->clone( osg::CopyOp::SHALLOW_COPY ) ) : new osgDB::ReaderWriter::Options;
    lopt->getDatabasePathList().push_front( osgDB::getFilePath( actualfilename ) );

    // remove the reader extension restriction in order to get the images too
    _acceptAll = true;
    return p_reader->readNode( input, lopt );
}

osgDB::ReaderWriter::ReadResult ReaderPhysFS::readObject( std::istream& fin, const osgDB::ReaderWriter::Options* options ) const
{
    assert( NULL && "not supported!" );
    return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
}

osgDB::ReaderWriter::ReadResult ReaderPhysFS::readImage( std::istream& fin, const osgDB::ReaderWriter::Options* options ) const
{
    assert( NULL && "not supported!" );
    return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
}

osgDB::ReaderWriter::ReadResult ReaderPhysFS::readNode( std::istream& fin, const osgDB::ReaderWriter::Options* options ) const
{
    assert( NULL && "not supported!" );
    return osgDB::ReaderWriter::ReadResult::ERROR_IN_READING_FILE;
}

//! Register the physfs plugin in osg db
REGISTER_OSGPLUGIN( physfs, ReaderPhysFS )

How to use the new plugin?

The physfs plugin can be integrated into an own application. It is also possible to make a dynamic library out of it like other osg plugins. In Yaf3D framework, the plugin is integrated into file system module. Here is an example for loading a mesh file using physfs:

    // read given file from virtual file system loader
    std::string fileName = "/mesh/mymesh.osg";
    osg::ref_ptr< osg::Node > node = osgDB::readNodeFile( fileName + ".physfs" );

Here is an example to load a font file in order to setup an osgText object:

    StreambufPtr streambuf = new Streambuf( fontpath );
    if ( !streambuf->ok() )
    {
        log_warning << "could not find font " << fontpath << " for player's floating text" << std::endl;
    }
    else
    {
        std::istream is( streambuf.getRef() );
        osg::ref_ptr < osgText::Font > font = osgText::readFontStream( is );
        ....
    }

Conclusion

This tutorial showed how to implement an osg plugin which allows loading files from a virtual file system. The code uses the PhysFS library and Yaf3D framework. The mechanism can be easily adapted to other virtual file system libraries than PhysFS, though.

For more details about PhysFS visit http://icculus.org/physfs

For more details about Yaf3D framework visit http://yag2002.sf.net or http://www.vr-fun.net

Cheers
boto