Gary Kratkiewicz BBN Technologies Geoffrey Knauth BAE Systems
description
Transcript of Gary Kratkiewicz BBN Technologies Geoffrey Knauth BAE Systems
Taming Development and Data Chaos in Military Logistics:
The ICIS Experience With Lightweight Languages
Gary Kratkiewicz BBN Technologies
Geoffrey KnauthBAE Systems
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
2
ICIS Background
• Web-based software system
• Analyzes future combat and humanitarian operations
• Measures ability of Defense Logistics Agency to satisfy wartime requirements for following items:• Fuel
• Food
• Replacement items (repair parts, clothing, medical items, etc.)
• Identifies potential problems• When will I run out of stock?
• What should I buy today for tomorrows needs?
• Do I have adequate sources & transportation?
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
3
Accomplishments
• Early Web-based DOD system
• Saved over $100 million • By computing reduced inventory needs for fuel and food
• Adopted as part of DOD doctrine
• Months of calculations reduced to hours• What-if capability leads to better decisions
• Short analyses expanded role to crisis action
• Used in military operations to:• Assess readiness
• Compute operational requirements
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
4
Development Notes
• Started in 1994
• Grew from small prototype to large system in 4 phases
• User interface moved to Web in stages from 1996 to 1999
• Approximately a third of code is in Lightweight Languages
• Significant data issues
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
5
Data, Data, Data
• Large amounts of data
• 800-1000 input files; 200 different formats
• Gigabytes of input and output per analysis
• Hundreds of analyses “active”
• Data management issues
• Received from many different agencies
• Regular and irregular updates
• Meandering formats
• Duplication between analyses: • Computations
• Intermediate & output files
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
6
1. Prototype ICIS – 1994/1995
• Original prototypes completely coded in Tcl/Tk
• Vague, rapidly changing, and conflicting requirements
• Less than 1 MB of data
• Ease of developing/tailoring lightweight prototypes greatly contributed to initial success
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
7
2. Production ICIS – 1995/1996
• Lots of data: approaching 1 GB
• Addition of (mostly) well-specified computational modules
• Real use with classified data in mid-1996
• Mix of languages:• Tcl/Tk for GUI and glue for module execution
• Perl for data preprocessing
• C/C++ for computational modules
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
8
3. Web ICIS – 1996-1999
• Address remote access issues
• Moved to Web in stages
• On server:• Tcl/Tk – glue for module execution
• Perl – data preprocessing
• C/C++ and Java – computational modules
• Perl and C++ – dynamic generation of HTML and JavaScript
• In browser:• HTML – static GUI elements
• JavaScript – form-based GUI elements
• Java AWT applets – interactive GUI elements
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
9
4. Redesigned Web ICIS – 2001/2002
• Improve GUI and better manage data• Data approaching 1 TB
• On server:• JScheme – managing data & module execution (sophisticated
“glue”)
• Perl – data preprocessing
• C/C++ and Java – computational modules
• In browser:• HTML – static GUI elements
• JavaScript (much less) – simple GUI elements
• Java AWT applets – more complex GUI elements
• Java Swing applets – most complex GUI elements
• JScheme for understanding scenario
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
10
Source Code Evolution
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
Prototype 1994 Production 1996 Some Web 1997 Full Web 1999 Redesign 2002
Lin
es o
f C
od
e
Tcl/Tk
C++
C
Perl
SQL
HTML
Java
Scheme
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
11
Summary of Lightweight Language Use in ICIS
• Rapid prototyping
• LL replaced by HL: completely for computations, partially for GUI
• Highly iterative, innovative design & development
• LL used in Data Set Manager, patent pending
• Processing of meandering data
• Gluing together heavyweight modules
• LL replaced with more sophisticated LL
• Building little languages
• Test code
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
12
JScheme — Application Flexibility
• Interpreter for debugging
• Scripts for component functionality
• Can be a server for execution management of multiple tasks
• Example implementation:
BBN’s Data Set Manager
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
13
User Asks ICIS Questions
• User interaction with ICIS is web-based
• User defines composite scenario
• User specifies results of interest
• ICIS runs the assessment
• Numerous modules might be run
• It might take a while…
• Results appear in browser
Example: “How much Fuel do I need?”
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
14
An Example Small Assessment
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
15
Problems
• Obvious approach of using shell scripts and files doesn’t cut it here
• Too much data
• Many computations recomputed needlessly
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
16
A Full ICIS AssessmentDoes Many Computations
The readable version of this graph takes up a large wall.
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
17
Data Set Manager
• Data is described by features, not file names
• A module spec describes
• Inputs and outputs of a module
• Process relationships with other modules
• Modules can be implemented in any language
• A module spec is applied to a composite scenario to produce an execution script
• The script is executed
• All computations are memoized
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
18
Execution Management Needs Dynamic Capabilities
Data Set Manager takes full advantage of Scheme
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
19
Implementing the Data Set Manager in JScheme was fairly easy because:
• Quasiquote was used to go from module spec to execution script
• A full language was required to write module specs
• Executing the script was just walking a tree structure
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
20
Murphy Strikes
Start
Goal
Mistrial!
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
21
MVC: JScheme = M+C, Java = V
Show DetailsTry Again
GenerateUser Options
Draw graph from knowledge of assessment state
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
22
Data Set Manager — Conclusion
• Core implementation is 3804 lines of JScheme code
• Supported by 287 lines of Perl, 102 lines of SQL
• ICIS Module specs add another 1853 lines of JScheme
• Current APIs for other languages are:
• 688 lines of C
• 496 lines of Perl
• 356 lines of Java (1 class)
• 120 lines of JScheme
• BBN has applied for two patents