Memory management benchmarks

One of the biggest problems facing memory management research has been the lack of a set of good benchmarks. While there is certainly more work to be done, we describe here the progress made and list some widely used (and proposed) benchmarks for memory management research for Java, C++ and Microsoft's CLI. We welcome suggestions for further benchmarks.

Java benchmarks

  • jvm98
  • jbb2000
  • jAppServer
  • pseudojbb
  • pBOB
  • Trade6
  • Colorado Bench
  • JOlden
  • Da Capo

    The benchmarks most widely used in the Java community for memory management research are those from the Standard Performance Evaluation Corporation (SPEC). The two SPEC suites most commonly used are jvm98 and jbb2000, but these have serious limitations. The most promising new development is the Da Capo benchmark suite, under development by a consortium of universities.

    jvm98 "allows users to evaluate performance for the combined hardware and software aspects of the JVM client platform". Evaluations of jvm98 can be found in

    Sylvia Dieckmann and Urs Hölzle. A study of the allocation behaviour of the SPECjvm98 Java benchmarks. in Eric Jul, editor, Proceedings of 12th European Conference on Object-Oriented Programming, ECOOP98, pages 92-115, volume 1445 of Lecture Notes in Computer Science, Brussels, July 1998. Springer-Verlag
    Sylvia Dieckmann and Urs Hölzle. The allocation behaviour of the SPECjvm98 Java benchmarks. In Rudolf Eigenman, editor, Performance Evaluation and Benchmarking with Realistic Applications, chapter 3, pages 77-108. MIT Press, 2001

    The chief drawbacks of the SPEC jvm98 suite are that only a few of the benchmarks perform substantial allocation activity (rather than, say, allocating a large array once and then manipulating its contents), and that none make serious use of threads.

    jbb2000 is a "benchmark for evaluating the performance of servers running typical Java business applications". It represents a multi-threaded order processing application for a wholesale supplier. Although the number of threads and the volume of data allocated is specified by the benchmark, this is tweakable by changing the SPECjbb.props file.

    One drawback for memory management is that SPEC jbb2000 reports the number of transactions executed in a fixed time. Many researchers therefore use Darko Stefanovic's pseudojbb which executes a fixed amount of work in a variable amount of time. A similar program appears to be available from here.

    jAppServer2004 s a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers. SPECjAppServer2004 is an end-to-end application which exercises all major J2EE technologies implemented by compliant application servers. It also heavily exercises all parts of the underlying infrastructure that make up the application environment, including hardware, JVM software, database software, JDBC drivers, and the system network.

    pBOB, IBM's Portable Business Object Benchmark, "provides a pure Java implementation of a typical transaction-oriented workload". Like jbb2000, pBOB is based on the TPC-C benchmark specification.

    Trade 6 is an IBM benchmark for the WebSphere Application Server.

    The Colorado Bench suite is a collection of 4 benchmarks: xalan, an XSLT processor, ipsixql, a persistent XML database, jigsaw, a webserver and nfc, a chatserver.

    The JOlden benchmarks are a Java re-write of the C pointer-intensive Olden benchmark. They are no longer recommended for performance measurements.

    The Da Capo group has recently released a beta-version of what is hoped to be a reliable benchmark suite for performance tests for the Java memory management community. The suite comprises a set of 11 non-trivial, real-world, open source Java benchmarks including the following programs:

    antlr parses one or more grammar files and generates a parser and lexical analyzer for each.
    batik renders a number of SVG files
    bloat performs a number of optimizations and analysis on Java bytecode files
    chart uses JFreeChart to plot a number of complex line graphs and renders them as PDF
    fop takes an XSL-FO file, parses it and formats it, generating a PDF file.
    hsqldb executes a JDBC-like in-memory benchmark, executing a number of transactions against a model of a banking application
    jython inteprets a series of Python programs
    pmd analyzes a set of Java classes for a range of source code problems
    ps reads and interprets a PostScript file
    xalan transforms XML documents into HTML
    The suite and a set of performance evaluation methodologies for dynamically compiled languages are described in Blackburn et al., OOPSLA 2006 The group welcomes suggestions for further benchmarks.

    C++ benchmarks

    Memory management benchmarks are much more problematic for C++ than for Java. C++ allows, and indeed many authorities encourage, users to write their own custom memory allocation routines (whether or not this is a good thing). Consequently, it is often difficult simply to plug in different memory managers in order to compare their performance. Here is a short list of available benchmarks:

    CLI benchmarks

    Good CLI benchmarks have been hard to come by. However, some repositories are starting to become available.

    Problems with this page?
    Contact the mm-net webmaster
    Last modified Friday 29 October 2004