Valerie Aurora ([info]valhenson) wrote,
@ 2008-04-01 23:11:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
File systems are irrelevant
I gave a lightning talk at the latest SuperHappyDevHouse designed to answer the question "What file system should I use?" for your typical web 2.0 startup. Answer? Make the file system irrelevant. Slides:

Storage for Startups

If you want the soundtrack to the slides, invite me to give a talk. :)

(Post a new comment)


[info]sire_maxus
2008-04-02 02:49 pm UTC (link)
thanks for the slides...

i have a question, maybe it's really basic or insane.. but web 2.0 isn't about community creation and modification? wouldn't that be much more writing to the file system compared to a normal web site?

thanks again and sorry if i'm asking something too insane..

P.S.: i friended you, i hope you don't mind

(Reply to this)

Web 2.0.. now with pictures.
[info]http://openid.aol.com/StephenSmoogen
2008-04-02 06:50 pm UTC (link)


Ohhh I like the drawings.. better than most I have to deal with. I will have to see the next time you give a talk in NM to hear the soundtrack. I also wonder what the long-winded version of this would be like :). [Ride of the Valkyries versus Goth dance]

The lecture got me to thinking more about some issues I had to deal with in a former life.. but its more on the "I have to store 480 gigs a day of data.." however the breaking the files down into smaller parts and 'hashing' them would probably have been an improvement over 240 2 gig files in one directory.:).

(Reply to this)

O(100G) does not mean what you think it does
[info]pavelmachek
2008-04-04 11:29 am UTC (link)
O(100G)==O(1), which probably was not what you wanted to say.

Plus, using hashed names on VFAT, with 10partitions per terabyte just to work around fs limitations seems pretty stupid, and management nightmare.

(Reply to this) (Thread)
Re: O(100G) does not mean what you think it does
[info]valhenson
2008-04-04 04:49 pm UTC (link)
Yet, everyone knows what I mean by O(100GB) instantly...

With regard to organizing data for your startup, it all depends on where you want your points of failure to be.

(Reply to this) (Parent)

[info]progrium
2008-04-15 07:07 am UTC (link)
Looks like you're on the SHDH front page now too...

(Reply to this)


[info]robbat2
2008-04-17 03:15 pm UTC (link)
MogileFS, on top of whatever you want.

(Reply to this)

XFS
(Anonymous)
2008-04-17 10:00 pm UTC (link)
Been using XFS for our latest projects/installs.

Seems better than all the we've tested (JFS,ReiserFS,EXT3) on RAID based SATA systems, and even better than ReiserFS on small+many file handling (which we didn't expect) without the large CPU overhead Reiser experiences.

Also creation times for an XFS drive are extremely fast. Not that it matters but..

Sequential reads/writes and delete performance of XFS is also outstanding.

Try tweaking your filesystem to works closer with how your hardware (controller) works, that usually does the trick. If you have much memory; try larger buffers. If you have many controllers; separate the logging from the data on different controllers (since transactional systems write both).

/Charlie Elgholm

(Reply to this)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…