realbasic-nug
[Top] [All Lists]

Re: Different behavior of Str(Double)? / Re: Floating point overflow

To: REALbasic NUG <realbasic-nug@lists.realsoftware.com>
Subject: Re: Different behavior of Str(Double)? / Re: Floating point overflow
From: Joe Strout <joe@inspiringapps.com>
Date: Tue, 30 Sep 2008 10:10:34 -0600
Authentication-results: mx.google.com; spf=neutral (google.com: 74.124.194.228 is neither permitted nor denied by best guess record for domain of realbasic-nug-bounces@lists.realsoftware.com) smtp.mail=realbasic-nug-bounces@lists.realsoftware.com
Delivered-to: listarchive@realsoftware.com
In-reply-to: <2CABC0FA-6EB3-4400-992C-E13156E153C8@great-white-software.com>
References: <mailman.1944.1222735349.3042.realbasic-nug@lists.realsoftware.com> <D8B903FF324D454BA0300ABE6555093D1757C5@BASEL.rothsoft.local> <2CABC0FA-6EB3-4400-992C-E13156E153C8@great-white-software.com>
Reply-to: REALbasic NUG <realbasic-nug@lists.realsoftware.com>
Sender: realbasic-nug-bounces@lists.realsoftware.com
On Sep 30, 2008, at 9:07 AM, Norman Palardy wrote:

2) format the number with the precision you want
tmp = format(number,",#.0000000") // exactly 7 digits after the decimal

That gives you 7 decimal places, not 7 significant digits. Getting a result with a desired number of significant digits is harder. (Off the top of my head, I suppose I might convert it to a string in exponential notation, then shift the decimal point manually...)

Best,
- Joe

--
Joe Strout
Inspiring Applications, Inc.
http://www.InspiringApps.com





_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives:
<http://support.realsoftware.com/listarchives/lists.html>


<Prev in Thread] Current Thread [Next in Thread>