Bunny Hudak (Moore)

25 Years of Innovation: The Life of Laurence P. Schermer

Posted Apr 24, 2003

By Henry Newman

This is a tribute to a close personal friend, co-worker, and technical genius who died tragically in April of 2003. I am writing this to honor his memory as well as show how one person can and has changed the technology we all use today.

Humble Beginnings

Laurence (Larry) P. Schermer began as a second shift customer engineer at Univac in 1977 after technical school and then completed a degree in math while working. In 1979, Larry moved to Cray Research, a highly innovative company that sought out equally innovative employees.

Larry's first job at Cray Research was to develop software for the unannounced Cray-1S I/O subsystem (IOS), so that it could read and write IBM 9 track tapes. Larry was part of a team that worked day and night for two straight years to develop software for:

• Booting the IOS
• Communicating with the Cray-1S system
• Disk drivers
• Most importantly, tape drivers and IBM tape emulation channels

This development was critical for Cray Research's strategy of selling to oil companies for seismic research and reservoir modeling, a market dominated by IBM because of tapes, not CPU performance. Without tape support on Cray systems, I believe that much of the oil found by these companies in the 1980s and early 1990s would not have been discovered, or the cost of finding it would have been significantly higher. Oil prices would likely be higher, which could have affected the entire economy. By the end of the 1980s, almost every major oil company in the world was using a Cray Research system.

UNIX and High Performance Computing

By the mid-1980s Cray had decided to move away from batch operating systems to a UNIX-based operating system. Many changes needed to be made to UNIX to support supercomputer functions. Larry helped develop functions such as:

• real memory
• A caching mechanism to take advantage of the Cray's SSD hardware
• The listio(3RT) system call, which conceptually was developed for the older Cray COS operating system

Each of these areas required Larry to make major modifications to the original UNIX source from AT&T. Some of these Larry did by himself, while others were done with the help of co-workers.

A major achievement for that time was a new file system that required:

• High performance I/O
• Round-robin allocation
• Separation of data and metadata
• Different allocations for large and small files
• Inodes that were 4K and could store data to reduce seek overhead

Every high performance file system today has either all or almost all of these features, and the same is especially true for the shared file systems. These features reduced the number of disk head seeks, allocation time, and data fragmentation, and also significantly improved performance. All of these innovations were developed largely by a single person for the Cray nc1fs in the late 1980s and are still used today.

Storage Area Networking Ahead of Its Time

Larry left Cray Research in 1992 and took a few years off as, like many of us who work in this industry have been at one time or another, he was burnt out. In 1994, he joined a company called NetStar (later purchased by Ascend and then Ascend by Lucent) that had an idea for a new router. At the time HiPPI (High Performance Parallel Interface) had become the predominant network interface in the HPC community for high speed networking given its peek rate of 100 MB/sec.

Larry worked with a hardware engineer to build a HiPPI to ATM converter. NetStar's goal was to have local HiPPI networking to WAN ATM connectivity. At the same time, Cray Research and other companies such as MaxStrat (a RAID company later purchased by Sun) were producing or going to produce HiPPI-based storage products. Cray had also developed a shared file system. Larry and I often discussed how it would be nice to access file system data without having to be local to the machine. The product for the most part was a market failure primarily due to HiPPI not being a commodity product and VERY expensive costs at the time for ATM WAN lines.

The key point, though, is that today we have commodity Fibre Channel directors from McData, Brocade, and Cisco that support WAN blades. We also have many products that support Fibre Channel to WAN connections from Nishan, LightSan, and others that support this concept. This originally was an idea that was developed in 1994 at NetStar.

RAID

In 1995, Larry again walked away from computing and took another year off. Early in 1996, Larry worked on a project to develop a HiPPI device driver for a new Fujitsu Supercomputer, the VPP700. Larry had never worked on the operating system, but as usual he dove in head first. Very early on he realized the performance of the HiPPI channel (~80 MB/sec data rate) would not be achievable from any user applications given the implementation of the MaxStrat Gen-5 RAID controller. Remember, this was 1996 -- very early in hardware RAID's life. The Gen-5 had a very small cache (24 MB) and allocated 64KB per device.

Given the backend structure of the GEN-5, the best performance from a LUN would be a 5+1 RAID-5 internally striped with another 5+1. 8+1 RAID-5 was not supported. Therefore, the stripe value would be 640KB (10*64KB). As almost all file systems at the time allocated in powers of 2, and HPC applications very often read and wrote in powers of 2, Larry immediately saw a huge problem. In his mind, the solution was simple -- add to the device drive a cache that would:

• Read/write on 640K boundaries to the RAID
• Readahead/writebehind in powers of 2 for the user requests and file system allocations
• Provide for syncing functions at shutdown or system crash

Adding this feature allowed the MaxStrat RAID to run at full rate for almost any sequential I/O access reading or writing. It eliminated almost all read-modify-write requests in the RAID and made better use of the very limited RAID cache, which was not a multiple of 640KB.

New File System and New Ideas

Because of the success of the MaxStrat HiPPI driver for Fujitsu, in 1997 Larry and I were asked to develop a new file system for the Fujitsu VPP5000. A number of innovations came from this project, many of which have appeared in other file systems. Here are some of the features of this file system, which was delivered in March of 1998 -- just 7 months after we started:

• Separate data and metadata and separate caches for each file system with different allocation sizes for various RAID alignments. With this feature you could align data for large RAID allocations with RAID-5 and align metadata for small allocation on RAID-1 
• Round-robin allocation for metadata. In cases where customers had a huge number of small files (especially common in some parts of the weather industry), the ability to have multiple metadata slices and different allocation sizes for metadata significantly improved performance for this type of environment 
• Variable length allocations up to 128MB (did not require powers of 2). This was for RAIDs that did not support power of 2 stripes 
• Metadata size could be up to 1MB. This was done so we could mkfs /usr file system. For example, where a metadata allocation equal to the largest command and read that command in without have to read the inode and read the data of the commands. This eliminated one disk read and associated head seek and missed revolution, significantly improving interactive response 
• A new allocation method that used allocation buckets and best fit instead of bitmaps and first fit and/or btrees and first fit. This eliminated data fragmentation to almost immeasurable amounts. 
• New fsck methodology. Given the overhead of logging file systems because metadata must be written two times, Larry believed that the requirement was fast fsck after reboot not logging. Instead of reading the file system meta blocks one at a time, Larry wrote an fsck that read the entire metadata device(s) and then processed it in memory. During our acceptance in 1998 with FW-SCSI RAID, we had to meet a requirement of being able to fsck 1 million files in a file system 7 directory levels in 20 seconds after a crash. We beat that by eight seconds, surpassing the acceptance criteria. I am unaware of any file system that uses this method even today.

In 2000, Larry worked at SRC, a company developing a hybrid machine using Intel processors and ASICs. In 2003, Larry took his final contract at a startup called Scale8. I am sure both of those companies benefited from the innovation, work, and, most importantly, mentoring that Larry provided to those co-workers just starting out

There are a number of unsung geniuses that have developed innovative technology we all use today. I have been lucky enough to know a few of them, and the industry just lost one of the best. I was fortunate to know him and even more fortunate to call him my friend.

Caroline Christian (Dunn)

Larry was always an easy-going guy. Often with a camera strap around his neck. When we were on the Campus Breeze staff, he showed me the process of developing film in the basement Art Room darkroom. Never pursued this hobby and true art form myself, but it was obvious that Larry loved the art of it plus the technical side. 

Marcia Soderholm

His work is way way more than I can understand. It's hard for me to follow. Lol. A good friend at the Hennepin County  was very close to him. Her husband met him at tech school. Larry helped him become an engineer. I know he was well loved by these dear friends. 

Jeffrey Walsh

Larry was a remarkable individual who showed you his true self.  I was fortunate to know him pretty well and spent lots of time in the summers going to North Mpls visiting him and likewise, Larry coming to my home in Seven corners, Mpls.  Larry would usually start my mornings with a full nelson to loosen me up for school and make sure I was awake.  He was probably one of the strongest people I've ever know.  When it came to his personality he was the kindest and most respectful of people.  He used to love my grandmother's slovak hamburgers which I would bring him one whenever she made them for lunch.  He offered me his peanut butter and banana sandwich which I usually gracefully declined.  Larry and I would go to movies and he like Elvis musicals and I like James Bond. He would always, without fail, sign my yearbook taking up the two inside pages with his message in the largest print you could imagine. What a guy. All in all, Larry was someone you could count on and was always helpful.  I remember after I broke my leg he would almost always be around to help me and loved pushing my wheel chair (sometimes faster than I was used to but safely). He would drive me to many of the class get togethers and made sure I was not going to miss anything. I attending his funeral in Highland Park which was a very sad day for me. He left us way too early but he is still in my memories as one of the best people I could have come across at U-High. TTFN

Bunny Hudak (Moore)

This is from Larry's brother Gene and sister Anita

They also sent me some pictures which I will put up in the Photo Gallery and on Larry's profile page. 

Thank you for your interest in learning more about Larry for the 50-year High School Reunion Memorial. 

After graduating high school Larry briefly attended St. Cloud State University. Subsequently, he transferred to the University of Minnesota where he enjoyed his time working on the Minnesota Daily.  It was important to Larry that his education has a purpose with a meaningful outcome.  He quickly learned that he wanted an education that actually prepared him to do something in life.  After about two years of college, uncertain about what he really wanted to do for a professional career, he left the University and spent time in Colorado where he found work on a small newspaper.   Shortly after, he returned to Minnesota and enrolled in Dunwoody where he took up electronics with an emphasis in computers.  Electronics and computers turned out to be the perfect match for Larry.   Upon leaving Dunwoody, he worked with Control Data, Univac, and Cray, with his first employer sending him back to the University to complete his College Degree.  Larry quickly reached a level of success, which gave him the freedom to work as a consultant, allowing him the opportunity and flexibility to pursue his true passions of travel and adventure. 

Larry loved to challenge himself mentally and physically.   He worked hard and played hard; often working 24/7 for weeks at a time, followed by hard play.   Larry had many interests, from computers to his adventures, photography, writing, and cooking and baking.  Everything he did, he put his all into – that was part of his strong work ethic.  He enjoyed a challenge and pursued it with gusto.  His work life took him around the US, and to Japan.  He climbed Mt. Rainier, completed the “Ride the Rockies” bike ride (a 450 mile ride through the Rockies), did some skydiving and scuba diving, and spent a vacation in Switzerland.   He balanced life with his artistic side, continuing to pursue his photography and writing classes at the loft. 

As strong as his work ethic, Larry was fiercely loyal to his family and friends-always there for you.   He had life long friends from school, and many of his closest friends and their families were also his and our family.  Larry lived his life to the fullest, and we all still miss him so much.

Thanks again,

Gene Schermer, U High class of ‘69

Anita Schermer