1st Edition

PRAGMATIC Security Metrics Applying Metametrics to Information Security

By W. Krag Brotby, Gary Hinson Copyright 2013
    512 Pages 63 B/W Illustrations
    by Auerbach Publications

    Other books on information security metrics discuss number theory and statistics in academic terms. Light on mathematics and heavy on utility, PRAGMATIC Security Metrics: Applying Metametrics to Information Security breaks the mold. This is the ultimate how-to-do-it guide for security metrics.

    Packed with time-saving tips, the book offers easy-to-follow guidance for those struggling with security metrics. Step by step, it clearly explains how to specify, develop, use, and maintain an information security measurement system (a comprehensive suite of metrics) to help:

    • Security professionals systematically improve information security, demonstrate the value they are adding, and gain management support for the things that need to be done
    • Management address previously unsolvable problems rationally, making critical decisions such as resource allocation and prioritization of security relative to other business activities
    • Stakeholders, both within and outside the organization, be assured that information security is being competently managed

    The PRAGMATIC approach lets you hone in on your problem areas and identify the few metrics that will generate real business value. The book:

    • Helps you figure out exactly what needs to be measured, how to measure it, and most importantly, why it needs to be measured
    • Scores and ranks more than 150 candidate security metrics to demonstrate the value of the PRAGMATIC method
    • Highlights security metrics that are widely used and recommended, yet turn out to be rather poor in practice
    • Describes innovative and flexible measurement approaches such as capability maturity metrics with continuous scales
    • Explains how to minimize both measurement and security risks using complementary metrics for greater assurance in critical areas such as governance and compliance

    In addition to its obvious utility in the information security realm, the PRAGMATIC approach, introduced for the first time in this book, has broader application across diverse fields of management including finance, human resources, engineering, and production—in fact any area that suffers a surplus of data but a deficit of useful information.

    Visit Security Metametrics. Security Metametrics supports the global community of professionals adopting the innovative techniques laid out in PRAGMATIC Security Metrics. If you, too, are struggling to make much sense of security metrics, or searching for better metrics to manage and improve information security, Security Metametrics is the place. http://securitymetametrics.com/

    Introduction
    Why Have We Written This Book?
    What’s Different about This Metrics Book?
    Who Are We Writing This For?
    Who Are We?
         Krag Brotby
         Gary Hinson
    What We’ll Be Talking About
    Defining Our Terminology
    What We Expect of You, the Reader
    Summary

    Why Measure Information Security?
    To Answer Awkward Management Questions
    To Improve Information Security, Systematically
    For Strategic, Tactical, and Operational Reasons
    For Compliance and Assurance Purposes
    To Fill the Vacuum Caused by Our Inability to Measure Security
    To Support the Information Security Manager
    For Profit!
    For Various Other Reasons
    Summary

    The Art and Science of Security Metrics
    Metrology, the Science of Measurement
    Governance and Management Metrics
    Information Security Metrics
    Financial Metrics (for Information Security)
    (Information Security) Risk Management Metrics
         Software Quality (and Security) Metrics
         Information Security Metrics Reference Sources
         Douglas Hubbard "How to Measure Anything" (Hubbard 2010)
         Andrew Jaquith: Security Metrics (Jaquith 2007)
          NIST SP 800-55: Performance Measurement Guide for Information Security (NIST 2008)
         Debra Herrmann: Complete Guide to Security and Privacy Metrics (Herrmann 2007)
         Krag Brotby: Information Security Management Metrics (Brotby 2009a)
         Lance Hayden: IT Security Metrics (Hayden 2010)
         Caroline Wong "Security Metrics: A Beginner’s Guide" (Wong 2012)
         ISO/IEC 27004: Information Security Management–Measurement (ISO/IEC 27004 2009) 3.7.9 CIS Security Metrics (CIS 2010)
         ISACA
    Specifying Metrics
    Metrics Catalogs and a Serious Warning About SMD
    Other (Information Security) Metrics Resources
    Summary

    Audiences for Security Metrics
    Metrics Audiences Within the Organization
         Senior Management
         Middle and Junior Management
         Security Operations
         Others with Interest in Information Security
    Metrics Audiences From Without the Organization
    Summary

    Finding Candidate Metrics
    Preexisting/Current Information Security Metrics
    Other Corporate Metrics
    Metrics Used in Other Fields and Organizations
    Information Security Metrics Reference Sources
    Other Sources of Inspiration for Security Metrics
         Security Surveys
         Vendor Reports and White Papers
         Security Software
    Roll-Your-Own Metrics
    Metrics Supply and Demand
    Summary

    Metametrics and the PRAGMATIC Approach
    Metametrics
    Selecting Information Security Metrics
    PRAGMATIC Criteria
    6.3.1 P = Predictive
    6.3.2 R = Relevant
    6.3.3 A = Actionable
    6.3.4 G = Genuine
    6.3.5 M = Meaningful
    6.3.6 A = Accurate
    6.3.7 T = Timely
    6.3.8 I = Independent
    6.3.9 C = Cost
    Scoring Information Security Metrics against the PRAGMATIC Criteria
    Other Uses for PRAGMATIC Metametrics
    Classifying Information Security Metrics
    6.6.1 Strategic/Managerial/Operational (SMO)Metrics Classification
    6.6.2 Risk/Control Metrics Classification
    6.6.3 Input–Process–Output (Outcome) Metrics Classification
    6.6.4 Effectiveness and Efficiency Metrics Classification
    6.6.5 Maturity Metrics Classification
    6.6.6 Directness Metrics Classification
    6.6.7 Robustness Metrics Classification
    6.6.8 Readiness Metrics Classification
    6.6.9 Policy/Practice Metrics Classification
    Summary

    150+ Example Security Metrics
    Information Security Risk Management Example Metrics
    Information Security Policy Example Metrics
    Security Governance, Management, and Organization Example Metrics
         Information Security Financial Management Metrics
         Information Security Control-Related Metrics
         Metrics for Business Alignment and Relevance of Controls
         Control Monitoring and Testing Metrics
         Financial Information Security Metrics
    Information Asset Management Example Metrics
    Human Resources Security Example Metrics
    Physical Security Examples
    IT Security Metric Examples
    Access Control Example Metrics
    Software Security Example Metrics
    Incident Management Example Metrics
    Business Continuity Management Examples
    Compliance and Assurance Metrics Examples
    Summary

    Designing PRAGMATIC Security Measurement System
    Brief History of Information Security Metrics
    Taking Systems Approach to Metrics
    Information Security Measurement System Lifecycle
    Summary

    Advanced Information Security Metrics
    High-Reliability Metrics
    Indicators and Proxies
    Key Indicators
         Key Goal Indicators (KGIs)
         Key Performance Indicators (KPIs)
         Key Risk Indicators (KRIs)
         Critical Success Factors (CSFs)
    Targets, Hurdles, Yardsticks, Goals, Objectives, Benchmarks, and Triggers
    Summary

    Downsides of Metrics
    Numbers Don’t Always Tell the Whole Story
    Scoring Political Points through Metrics
    Implausible Deniability
    Metrics Gaps
    On Being Good Enough
    What Not to Measure
    Summary

    Using PRAGMATIC Metrics in Practice
    Gathering Raw Data
         Sampling
         Automated Data Sources
         Observations, Surveys, and Interviews
         Online or In-Person Surveys
         Scoring Scales
         Audits, Reviews, and Studies
    Data Analysis and Statistics
    Data Presentation
         General Considerations
         Analytical Tools and Techniques
         Reporting Tools and Techniques
         Presentational Tools and Techniques
         Graphs, Figures, Diagrams, and Illustrations
         Drawing Attention to Specific Issues
    Using, Reacting to, and Responding to Metrics
         Periodic versus Event-Driven Reporting
    Summary

    Case Study
    The Context: Acme Enterprises, Inc.
    Information Security Metrics for C-Suite
         Information Security Metrics for the CEO
         Information Security Metrics for the CIO
         Information Security Metrics for the CISO
         Information Security Metrics for the CFO
         Information Security Metrics for the VP of Production
         Information Security Metrics for the VP of Marketing
    Information Security Metrics for Management and Operations
    Information Security Metrics for External Stakeholders
    Acme’s Information Security Measurement System
    Summary

    Conclusions
    Take-Home Lessons from This Book
         On Pragmatism and Being PRAGMATIC
         On Giving You the Confidence and Skills to Have a Go
         On Improving the Quality of Your Management Information through Metametrics
         On Improving Metrics of All Sorts
    Your Chance to Advance the Profession and the Practice of Metrics
    An Action Plan to Take Away
    Summary

    Appendix A: PRAGMATIC Criteria
    Appendix B: Business Model of Information Security (BMIS)
    Appendix C: Capability Maturity Model (CMM)
         Level 1–Initial
         Level 2–Repeatable
         Level 3–Defined
         Level 4–Managed
         Level 5–Optimizing
    Appendix D: Example Opinion Survey Form
         Security Awareness Survey on Malware
    Appendix E: SABSA Security Attributes Table
    Appendix F: Prototype Metrics Catalog
    Appendix G: Effect of Weighting the PRAGMATIC Criteria
    Appendix H: ISO27k Maturity Scale Metrics
    Appendix I: Sample Management Survey
    Appendix J: Observer Bias
    Appendix K: Observer Calibration
    Appendix L: Bibliography

    Biography

    Krag Brotby has 30 years of experience in the area of enterprise computer security architecture, governance, risk, and metrics and is a Certified Information Security Manager (CISM) and Certified in the Governance of Enterprise Information Technology qualifications. Krag is a CISM trainer and has developed a number of related courses in governance, metrics, governance-risk-compliance (GRC), and risk and trained thousands on five continents during the past decade.
    Krag's experience includes intensive involvement in current and emerging security architectures, IT and information security metrics, and governance. He holds a foundation patent for digital rights management and has published a variety of technical and IT security-related articles and books. Brotby has served as principal author and editor of the Certified Information Security Manager Review Manual (ISACA 2012) since 2005, and is the researcher and author of the widely circulated Information Security Governance: Guidance for Boards of Directors and Executive Management (ITGI 2006), and Information Security Governance: Guidance for Information Security Managers (ITGI 2008a) as well as a new approach to Information Security Management Metrics (Brotby 2009a) and Information Security Governance; A Practical Development and Implementation Approach (Brotby 2009b). Krag has served on ISACA's Security Practice Development Committee. He was appointed to the Test Enhancement Committee, responsible for testing development, and to the committee developing a systems approach to information security called the Business Model for Information Security (BMIS). He received the 2009 ISACA John W. Lainhart IV Common Body of Knowledge Award for noteworthy contributions to the information security body of knowledge for the benefit of the global information security community.

    Krag is a member

    Like all books on metrics, PRAGMATIC Security Metrics: Applying Metametrics to Information Security makes the statement that "you can't manage what you can't measure".  The authors claim that other books on information security metrics discuss number theory and statistics in academic terms. This title promises to be light on mathematics and heavy on utility and is meant as a how-to-do-it guide for security metrics.

    As to the title, PRAGMATIC is an acronym for the basis of the method of the book, in using metrics that are predictive, relevant, actionable, genuine, meaningful, timely, independent and cost. After reading the first chapter, PRAGMATIC Security Metrics: Applying Metametrics to Information Security looks like it may live up to its promise of being able to use metrics not only to track and report performance but to identify problem areas and opportunities, and drive information security improvements. If so, this could be the metrics book a lot of information security professionals have been waiting for.
    —Ben Rothke, CISSP, CISM, Information Security Manager, Wyndham Worldwide; and author of Computer Security: 20 Things Every Employee Should Know, writing on the RSA Conference Blog, www.rsaconference.com