Mulgara Project: Issueshttps://code.mulgara.org/https://code.mulgara.org/favicon.ico?15861924492008-10-23T13:22:20ZMulgara Project
Redmine Mulgara - Feature #160 (Closed): Cache lucene indexeshttps://code.mulgara.org/issues/1602008-10-23T13:22:20Zronald -ronald@foo.bar
<p>The lucene indexes in the [<a class="source" href="https://code.mulgara.org/projects/mulgara/repository/3/entry/trunk/src/jar/resolver-lucene">source:trunk/src/jar/resolver-lucene</a>/ lucene resolver]<br />are created anew for each transaction; this is fairly costly. We should cached them<br />across transactions, re-opening them if a modification occurred since the last time<br />they were checked out.</p> Mulgara - Feature #158 (Closed): Cache index readers and writers in !LuceneResolverhttps://code.mulgara.org/issues/1582008-10-19T10:14:54Zronald -ronald@foo.bar
<p>New lucene index readers and writers are currently created for every<br />transaction. This is costly, so we should provide a pool of index readers<br />and writers, and re-open the readers at the start of a transaction if<br />anything has been written since they were last (re-)opened.</p> Mulgara - Bug #133 (Closed): Sporadic concurrent access in StringPoolSessionhttps://code.mulgara.org/issues/1332008-07-24T07:27:07Zronald -ronald@foo.bar
<p>Twice, on two different machines/os's/jdk's we've seen the<br />ItqlInterpreterBeanUnitTest fail with the following stack trace:<br /><pre>
org.mulgara.query.QueryException: Unable to load file:/Users/pag/src/mulgara/trunk/data/camera.n3 into rmi://10.0.1.198/server1#itqlmodel
at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:726)
at org.mulgara.resolver.DatabaseSession.setModel(DatabaseSession.java:562)
at org.mulgara.server.rmi.SessionWrapperRemoteSession.setModel(SessionWrapperRemoteSession.java:123)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at org.mulgara.server.rmi.RemoteSessionImpl_Stub.setModel(Unknown Source)
at org.mulgara.server.rmi.RemoteSessionWrapperSession.setModel(RemoteSessionWrapperSession.java:166)
at org.mulgara.query.operation.Load.doTx(Load.java:101)
at org.mulgara.query.operation.DataInputTx.sendMarshalledData(DataInputTx.java:111)
at org.mulgara.query.operation.Load.execute(Load.java:79)
at org.mulgara.itql.ItqlInterpreterBean.load(ItqlInterpreterBean.java:707)
at org.mulgara.itql.ItqlInterpreterBeanUnitTest.testLoadApi8(ItqlInterpreterBeanUnitTest.java:577)
Caused by: org.mulgara.query.QueryException: org.mulgara.query.MulgaraTransactionException: Transaction rollback triggered
at org.mulgara.resolver.MulgaraInternalTransaction.implicitRollback(MulgaraInternalTransaction.java:516)
at org.mulgara.resolver.MulgaraInternalTransaction.execute(MulgaraInternalTransaction.java:627)
at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:723)
at org.mulgara.resolver.DatabaseSession.setModel(DatabaseSession.java:562)
at org.mulgara.server.rmi.SessionWrapperRemoteSession.setModel(SessionWrapperRemoteSession.java:123)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
Caused by: org.mulgara.query.QueryException: org.mulgara.resolver.spi.ResolverException: Unable to read input statements
at org.mulgara.resolver.store.StatementStoreResolver.modifyModel(StatementStoreResolver.java:381)
at org.mulgara.resolver.InternalResolver.modifyModel(InternalResolver.java:164)
at org.mulgara.resolver.SetModelOperation.execute(SetModelOperation.java:153)
at org.mulgara.resolver.MulgaraInternalTransaction.execute(MulgaraInternalTransaction.java:623)
Caused by: org.mulgara.query.QueryException: org.mulgara.query.TuplesException: Exception while reading file:/Users/pag/src/mulgara/trunk/data/camera.n3
at org.mulgara.content.n3.Parser.checkForException(Parser.java:517)
at org.mulgara.content.n3.Parser.getTriple(Parser.java:539)
at org.mulgara.content.n3.N3Statements.next(N3Statements.java:389)
at org.mulgara.resolver.store.StatementStoreResolver.modifyModel(StatementStoreResolver.java:353)
Caused by: java.lang.IllegalStateException: Concurrent Access of [[StringPoolSession]] Attempted
at org.mulgara.resolver.StringPoolSession.checkCurrentThread(StringPoolSession.java:787)
at org.mulgara.resolver.StringPoolSession.localizePersistent(StringPoolSession.java:207)
at org.mulgara.resolver.store.StatementStoreResolver.localizePersistent(StatementStoreResolver.java:451)
at org.mulgara.resolver.PersistentResolverSession.localize(PersistentResolverSession.java:97)
at org.mulgara.content.n3.Parser.createBlankNode(Parser.java:453)
at org.mulgara.content.n3.Parser.getBlankNode(Parser.java:424)
at org.mulgara.content.n3.Parser.toNode(Parser.java:362)
at org.mulgara.content.n3.Parser.quad(Parser.java:272)
at com.hp.hpl.jena.n3.N3AntlrParser.emitQuad(N3AntlrParser.java:66)
at com.hp.hpl.jena.n3.N3AntlrParser.objectList(N3AntlrParser.java:738)
at com.hp.hpl.jena.n3.N3AntlrParser.propValue(N3AntlrParser.java:622)
at com.hp.hpl.jena.n3.N3AntlrParser.propertyList(N3AntlrParser.java:336)
at com.hp.hpl.jena.n3.N3AntlrParser.statement0(N3AntlrParser.java:287)
at com.hp.hpl.jena.n3.N3AntlrParser.statement(N3AntlrParser.java:204)
at com.hp.hpl.jena.n3.N3AntlrParser.document(N3AntlrParser.java:153)
at com.hp.hpl.jena.n3.N3Parser.parse(N3Parser.java:61)
at org.mulgara.content.n3.Parser.run(Parser.java:213)
</pre></p>
<p>Unfortunately, this has not been reproduceable. But interestingly<br />enough the stack trace was identical in both cases.</p> Mulgara - Feature #47 (Closed): (Patch) query speed improvement by 30% - 40%https://code.mulgara.org/issues/472007-02-26T12:31:17Zronald -ronald@foo.bar
<pre>
<code class="html syntaxhl">Creating exceptions (instances of Throwable) is an expensive
<span class="nt"><br/></span>
operation, mostly because of the stack trace. However, browsing the
<span class="nt"><br/></span>
code I've seen a number of places where 'new Throwable()' was used
<span class="nt"><br/></span>
unconditionally. Therefore I ran some experiments.
<span class="nt"><br/></span>
<span class="nt"><br/></span>
Setup: a db with 1.4 million triples (production data), and 200
<span class="nt"><br/></span>
<span class="ni">&quot;</span>production<span class="ni">&quot;</span> queries (most of them unique). Each test was run by
<span class="nt"><br/></span>
starting mulgara, running the first 100 queries, then timing the
<span class="nt"><br/></span>
second 100 queries, and finally shutting mulgara down again. The
<span class="nt"><br/></span>
log level was WARN, so that normally nothing gets printed. No
<span class="nt"><br/></span>
explicit transactions were used (i.e. autoCommit=true).
<span class="nt"><br/></span>
<span class="nt"><br/></span>
Some queries were simple (on the order of 50 ms or less to answer),
<span class="nt"><br/></span>
some more complex (close to a second); the client was using SOAP to
<span class="nt"><br/></span>
access the db, which introduces typically between 40 and 100 ms
<span class="nt"><br/></span>
overhead per query, so these results are <span class="ni">&quot;</span>worst case<span class="ni">&quot;</span>, i.e. the
<span class="nt"><br/></span>
speed-up noted is probably slightly higher using RMI.
<span class="nt"><br/></span>
<span class="nt"><br/></span>
Three sets of tests were run: straight rev <span class="nt"><a</span> <span class="na">href=</span><span class="s">"http://mulgara.org/trac/changeset/192"</span><span class="nt">></span>192<span class="nt"></a></span> (<span class="ni">&quot;</span>with stack traces<span class="ni">&quot;</span>),
<span class="nt"><br/></span>
a version with all 'new Throwable()' removed (<span class="ni">&quot;</span>without stack traces<span class="ni">&quot;</span>),
<span class="nt"><br/></span>
and a version with various 'new Throwable()' surounded by appropriate
<span class="nt"><br/></span>
conditions (<span class="ni">&quot;</span>partial stack traces<span class="ni">&quot;</span>); the patch for this last set is
<span class="nt"><br/></span>
the one attached to this issue.
<span class="nt"><br/></span>
<span class="nt"><br/></span>
Each test was run with 1, 2, and 3 client threads hitting the db.
<span class="nt"><br/></span>
<span class="nt"><br/></span>
Here are the results (the alignment of the table will probably be
<span class="nt"><br/></span>
messed up in the html):
<span class="nt"><br/></span>
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>stack-traces threads times (sec) avg min avg% min%
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>with 1 29.2 31.4 29.0 29.9 29.0 100 100
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>with 2 18.7 19.5 18.1 18.8 18.1 100 100
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>with 3 17.7 17.6 19.2 18.2 17.6 100 100
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>without 1 17.7 17.7 18.1 17.8 17.7 60 61
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>without 2 13.0 12.5 13.1 12.9 12.5 69 69
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>without 3 12.6 12.2 12.5 12.4 12.2 68 69
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>partial 1 18.0 18.6 18.0 18.2 18.0 61 62
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>partial 2 12.5 12.9 13.3 12.9 12.5 69 69
<span class="nt"><br/></span>
<span class="ni">&nbsp;&nbsp;</span>partial 3 12.8 11.2 13.1 12.4 11.2 68 64
<span class="nt"><br/></span>
<span class="nt"><br/></span>
As you can see, the <span class="ni">&quot;</span>partial<span class="ni">&quot;</span> (which is the attached patch) gives
<span class="nt"><br/></span>
around 30% - 40% reduction in query time.
<span class="nt"><br/></span>
<span class="nt"><br/></span>
The attached patch is quite simple and quite safe. In a couple
<span class="nt"><br/></span>
cases it makes the Throwable instantiation conditional on the log
<span class="nt"><br/></span>
level at which it will get printed, and in the other two cases it
<span class="nt"><br/></span>
removes unnecessary (i.e. duplicate) stack-trace creation. In
<span class="nt"><br/></span>
short it does not remove any information from the logs.
<span class="nt"><br/></span>
<span class="nt"><br/></span>
No tests are affected by this patch.
<span class="nt"><br/></span>
</code></pre>