1. 03 Nov, 2016 1 commit
  2. 01 Nov, 2016 2 commits
    • Jason Rhinelander's avatar
      crdata: fix thread count problem · c8ca4f27
      Jason Rhinelander authored
      Exiting in preload mode wouldn't trigger the thread count decrement;
      fixed it by breaking the while loop instead of returning directly.
      c8ca4f27
    • Jason Rhinelander's avatar
      creativity-data: fix prctl status · 8ee7e194
      Jason Rhinelander authored
      prctl needs to run in the main thread to show up in top (since top
      shows the main thread's name when not showing individual threads),
      this commit adds machinery to handle that via a wait loop in the main
      thread.
      8ee7e194
  3. 31 Oct, 2016 1 commit
  4. 27 Oct, 2016 3 commits
    • Jason Rhinelander's avatar
      Removed debugging · fc9ea94f
      Jason Rhinelander authored
      fc9ea94f
    • Jason Rhinelander's avatar
      gcc-4.9 fix · 68e29e88
      Jason Rhinelander authored
      Sadly, libstdc++ doesn't support moving a stringstream before v5, so
      work around it by accepting a unique_ptr instead.
      68e29e88
    • Jason Rhinelander's avatar
      creativity-data: added file preloading · bc62d7ce
      Jason Rhinelander authored
      Right now processing is limited by the fact that individual threads are
      trying to read/decompress different files at once, and don't read from
      disk until they start working.  By using a single thread to load files
      from disk into memory, then having individual worker threads load from
      memory, the disk contention bottleneck should be reduced: one process
      ends up reading files from disk as fast as possible, but sequentially,
      and so while the disk may still bottleneck the process (particular if
      the disk is slow or the CPUs are numerous and very fast), that
      bottleneck should be reduced by reading everything sequentially in one
      thread.
      bc62d7ce
  5. 18 Sep, 2016 1 commit
    • Jason Rhinelander's avatar
      Generalized policies; add catch-and-fine policy stub · a76ec21a
      Jason Rhinelander authored
      This commit reworks the public_sharing settings and terminology into
      more generic "policy" settings and terminology, and adds support for
      enabling a new "catch-pirates" policy.
      
      (catch-pirates is not yet implemented, and is protected by an exception
      if used, but will come soon).
      a76ec21a
  6. 24 Jun, 2016 1 commit
    • Jason Rhinelander's avatar
      Use eris::Serialization for data storage · f7672169
      Jason Rhinelander authored
      Most of the nuissance code (block lists, compression, header value
      handling) moved to eris::Serialization; what's left is considerably
      simpler.
      
      XZ compression is now always enabled; removed the option to enable it.
      
      The memory-all option is gone now; with xz compression always enabled,
      it didn't do anything --memory-xz didn't already do (unless you
      actually unxz a file manually and pass it in).
      
      Added --memory CLI option to creativity-gui to let it use more memory
      instead of tmpfiles when running.
      
      Added --tmpdir to various cli tools that didn't already have it; it gets
      used during decompression (when not using --memory-xz/--memory).
      
      Converted BookState values to uint32_t instead of unsigned int.  They
      are probably the same, but being explicit makes sure I'm writing 32-bit
      ints in the data file.
      
      The big-endian/little-endian code is gone (it's now in the
      eris serializer<T> code).
      
      The big mess of constant header locations is gone.  It was a nuissance
      anyway--the new approach is to just register them all in a special
      method, and let Serializer remember which one goes where.
      
      Use full eris_id_t for stored ids.  Originally I truncated these to
      32-bit ints to save space, but with compression enabled that really
      isn't saving much.  It also simplifies the code a bit, by being able to
      serialized eris_id_t values directly without a type cast to uint32_t.
      
      The number of states is no longer stored as a header; it's now just the
      size of the state locations pointer block, which comes right after the
      header.
      
      The reader-libraries outer list is now written when creating a new
      file; previously it was delayed until the first state gets added.  The
      inner list (each reader's library) get created as needed.
      
      Reordered methods in FileStorage.cpp to put the
      readWhatever/writeWhatever pairs adjacent to each other, which makes it
      a bit easier to see that they are reading/writing the same things.
      f7672169
  7. 05 Feb, 2016 1 commit
    • Jason Rhinelander's avatar
      Fix buggy min # periods checking · aa484224
      Jason Rhinelander authored
      The logic for figuring out whether to skip a file was screwed up,
      resulting in failure (either segfault or out of range exceptions) if
      called on a file that had too few periods.
      aa484224
  8. 03 Feb, 2016 2 commits
    • Jason Rhinelander's avatar
      Fix storage bug: don't hold Creativity shared ptr · a39e0382
      Jason Rhinelander authored
      Reader, Book, etc. stored shared_ptrs to the Creativity object owning
      their simulation, but this means:
      
      Creativity -> Simulation -> Reader
           ^------------------------'
      
      and so resetting the Creativity shared_ptr (in creativity->cli) wasn't
      actually deleting the object.
      
      This commit changes all the members (Reader, Book, etc.) to store a hard
      reference (either Creativity& or const Creativity&)--since the members
      belong to the simulation, which is destroyed during the Creativity
      destruction.
      
      This was causing a serious error in storage, because the lack of
      destruction meant the file wasn't closed.  This was compounded by
      flush_for not actually doing a file handle flush (since that was
      handled in FileStorage's flush override), though that has been fixed in
      the previous commit.
      
      With that gone, there really is no need to force Creativity access
      through a shared_ptr, and so this commit also makes Creativity used as
      a regular class just about everywhere (except cli, which uses a
      unique_ptr--because it needs to control destruction).
      a39e0382
    • Jason Rhinelander's avatar
      series/quantiles: source-based quantile support · 99de1bbf
      Jason Rhinelander authored
      Old behaviour:
      ==============
      
      creativity-series previously removed non-finite values, and sorted the
      rest, producing something like:
      
      t,1st,2nd,3rd,...
      2,10,11,12
      3,3
      
      (where t=0 and t=1 have no finite values, t=3 has only one non-finite,
      etc.).  This discards any association with the files they come from,
      however, which means any sort of source-based confidence exclusion is
      impossible.
      
      creativity-series-quantiles then used these pre-sorted values to produce
      another .csv of quantile values.
      
      createivity-series-graphs accepted either one--if the series file, it
      calculated quantiles on the fly; if the quantiles, it used the
      pre-calculated quantiles.
      
      New behaviour:
      ==============
      
      creativity-series now just generates something like:
      
      t,source1,source2,source3
      0,nan,nan,nan
      1,nan,nan,nan
      1,12,10,11
      2,nan,3,nan
      
      which creativity-series-quantiles and creativity-series-graphs now
      understand: they does the nan trimming and sorting, then calculate
      quantiles.
      
      creativity-series-graphs also gains an entirely new ability, with flag
      --source-confidence (but only when given a creativity-series file): to
      select a confidence region by excluding (1-x)% of source files, then
      plotting the minimum and maximum values remaining after removal of the
      most non-median values.  This is done by calculating (p-0.5)^2 for the
      inverse quantile p for each time period, then summing these up across
      time periods: the source files with the largest scores are excluded.
      (see --help for the gory details).
      99de1bbf
  9. 11 Dec, 2015 1 commit
    • Jason Rhinelander's avatar
      data/Data.* -> data/simdata.*; added data/util.hpp · 0f86ee11
      Jason Rhinelander authored
      Renamed Data.hpp to simdata.hpp since it never contained a "Data" class
      (so keep with the lower-case-for-non-classes naming convention).
      
      Moved the data utility functions (double to string, quantile, and
      variance) to the new util.hpp from data.cpp and
      creativity/data/simdata.hpp
      0f86ee11
  10. 12 Oct, 2015 1 commit
  11. 10 Oct, 2015 1 commit
    • Jason Rhinelander's avatar
      Lock output around status message · eb4b5a35
      Jason Rhinelander authored
      Prevents multiple threads trying to write the current file at once
      (this basically only appears at startup; two threads finishing at
      identical times is otherwise vanishingly unlikely).
      eb4b5a35
  12. 09 Oct, 2015 1 commit
    • Jason Rhinelander's avatar
      quality and scale quantiles; short-run stats · d20ef983
      Jason Rhinelander authored
      Added percentiles for (5,10,25,50,75,90,95%) in calculated data for
      quality and creator scale factor.
      
      Short-run statistics--that is, for the *first* T piracy/public
      periods--are now calculated (prefixed with piracy.SR. instead of
      piracy.).
      
      Also adds creation_fixed parameter.
      
      Updated human-readable output to use . instead of _ to match CSV fields.
      d20ef983
  13. 30 Sep, 2015 2 commits
  14. 21 Sep, 2015 4 commits
    • Jason Rhinelander's avatar
      Threaded data compilation · 74cf3bb6
      Jason Rhinelander authored
      Added a threads option to load and process each file in a thread.  This
      increases performance considerably since the processing is CPU bound
      rather than IO bound.
      
      Performance processing 8 files at -j4 seems to be about 2.5x performance
      at -j0, with caches cleared loading from my office computer SSD.
      Loading from a 1.5TB WD Green HDD (with caches cleared), performance
      improves by less: about 1.5x, but still an improvement.
      74cf3bb6
    • Jason Rhinelander's avatar
      Don't overcache states, to save memory · 75387b25
      Jason Rhinelander authored
      Right now all 25 pre, 25 piracy, and 25 public states get loaded, but
      we only need 25 of these at a time, so clear out the cache between
      pre/piracy and piracy/public to reduce memory usage.
      75387b25
    • Jason Rhinelander's avatar
      Undo threaded data commit: it was slower · 9fa68aaa
      Jason Rhinelander authored
      The threaded code of the previous commit was slightly slower, so
      removing it.
      9fa68aaa
    • Jason Rhinelander's avatar
      Use a thread to parse piracy/public files · 393331ec
      Jason Rhinelander authored
      Testing this to see if it's worthwhile.
      393331ec
  15. 17 Sep, 2015 1 commit
    • Jason Rhinelander's avatar
      Reduce precision when possible · 62c7595d
      Jason Rhinelander authored
      Reduce precision by one or two digits when doing so doesn't result in
      numerical loss; this avoids outputting things like 0.1000...001 instead
      of just 0.1, but still yields 0.1000...002 when the last digit makes a
      difference for the round trip conversion back to double.
      62c7595d
  16. 14 Sep, 2015 1 commit
  17. 13 Sep, 2015 1 commit
    • Jason Rhinelander's avatar
      Added preliminary results parser · 607294dc
      Jason Rhinelander authored
      creativity-results takes the CSV results as produced by
      creativity-data, and will (once finished) produce tabulated output of
      the relationships in the variables.
      
      Currently this parses the CSV files (which were also simplified a bit,
      to eliminate things like quoting, which we really don't need here).
      Next is to use SUR to actually analyze that data.
      607294dc
  18. 29 Aug, 2015 1 commit
  19. 28 Aug, 2015 1 commit
  20. 04 Feb, 2015 1 commit
  21. 02 Feb, 2015 1 commit
    • Jason Rhinelander's avatar
      Simplified arguments, added piracy skip periods · 2e323546
      Jason Rhinelander authored
      It doesn't make a whole lot of sense to NOT use the same value for the
      previous -t, -p, -z arguments, so just collapsed them into a single
      -t/--periods argument.
      
      Also added a --skip-piracy argument, that lets the "new piracy" period
      begin after a certain number of periods.  This helps avoid huge
      mispredictions coming in t=P when piracy first becomes available: most
      simulations seem to have significant utility hits there.
      2e323546
  22. 01 Feb, 2015 1 commit
  23. 30 Jan, 2015 1 commit
  24. 27 Jan, 2015 1 commit
    • Jason Rhinelander's avatar
      Removed postgresql support. · 6b1323b2
      Jason Rhinelander authored
      It worked, but was considerably less efficient than file storage.
      Removing it because keeping it updated for various other changes is a
      nuissance.
      6b1323b2
  25. 10 Jan, 2015 1 commit
    • Jason Rhinelander's avatar
      Added creativity-data script · c2815465
      Jason Rhinelander authored
      Takes one or more results files and calculates a bunch of statistics,
      outputting a CSV result of the input files with one row per result file.
      c2815465