Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [EXC-1768] Add system API to get costs of management canister calls. #3584

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
9c07654
SystemAPI trait
michael-weigelt Jan 23, 2025
77c3096
SystemAPI trait changes
michael-weigelt Jan 28, 2025
14e8d66
implement key costs
michael-weigelt Jan 28, 2025
ba7f100
implement replication factor
michael-weigelt Jan 28, 2025
c18d911
implement cost_create_canister
michael-weigelt Jan 28, 2025
14e84a5
implement cost_call
michael-weigelt Jan 28, 2025
c4298fb
implement cost_http_request
michael-weigelt Jan 29, 2025
ed95b9a
fix ssss
michael-weigelt Jan 29, 2025
8613b76
clippy
michael-weigelt Jan 29, 2025
b50c819
clippy
michael-weigelt Jan 29, 2025
18a6aec
added signatures
michael-weigelt Jan 29, 2025
e087737
added replication factor linker
michael-weigelt Jan 29, 2025
f26f2bb
implement other linkers
michael-weigelt Jan 29, 2025
5e6c8b9
benchmarks
michael-weigelt Jan 29, 2025
552827c
enums
michael-weigelt Jan 29, 2025
bc3edbe
api availability
michael-weigelt Jan 29, 2025
63ac1cd
Merge remote-tracking branch 'origin' into mwe/costs
michael-weigelt Jan 29, 2025
a69255b
Update rs/interfaces/src/execution_environment.rs
michael-weigelt Feb 3, 2025
fcc7e41
Update rs/interfaces/src/execution_environment.rs
michael-weigelt Feb 3, 2025
1ff90d2
use full keyid for lookup
michael-weigelt Feb 4, 2025
a213b00
Merge remote-tracking branch 'origin' into mwe/costs
michael-weigelt Feb 4, 2025
f0f131f
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 4, 2025
b629e57
tryinto curves
michael-weigelt Feb 4, 2025
e3f6a0a
tryinto curves
michael-weigelt Feb 4, 2025
47e7ad5
conversion in system_api, not embedders
michael-weigelt Feb 4, 2025
936d0f0
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 4, 2025
81b8b70
Update rs/types/management_canister_types/src/lib.rs
michael-weigelt Feb 4, 2025
0b9a95e
Update rs/types/types/src/consensus/idkg/common.rs
michael-weigelt Feb 4, 2025
f9b5f5c
import
michael-weigelt Feb 4, 2025
8bb8a36
Merge branch 'mwe/costs', remote-tracking branch 'origin' into mwe/co…
michael-weigelt Feb 4, 2025
600d6ef
Merge remote-tracking branch 'origin' into mwe/costs
michael-weigelt Feb 4, 2025
b7825cd
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 4, 2025
1c7e712
validation
michael-weigelt Feb 4, 2025
e33abab
linkers
michael-weigelt Feb 4, 2025
035c113
benchmarks
michael-weigelt Feb 4, 2025
108dedc
naming
michael-weigelt Feb 4, 2025
f2c19f6
universal canister
michael-weigelt Feb 4, 2025
9ac4d8f
universal canister 2
michael-weigelt Feb 4, 2025
dbab533
copy paste error
michael-weigelt Feb 5, 2025
9bb4c65
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 5, 2025
588dc4b
fix tests
michael-weigelt Feb 5, 2025
c23811b
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 5, 2025
47df030
uni canister 3
michael-weigelt Feb 5, 2025
a649795
fix universal canister + test
michael-weigelt Feb 5, 2025
23d6bcb
Revert "implement replication factor"
michael-weigelt Feb 10, 2025
652b616
return types for cost api
michael-weigelt Feb 10, 2025
d44ca2d
Merge branch 'mwe/costs', remote-tracking branch 'origin' into mwe/co…
michael-weigelt Feb 10, 2025
f4a45d7
return values for sign cost endpoints
michael-weigelt Feb 10, 2025
419d290
Merge remote-tracking branch 'origin' into mwe/costs
michael-weigelt Feb 10, 2025
e633b83
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 10, 2025
3f20c23
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 10, 2025
12242a5
change sign signatures
michael-weigelt Feb 10, 2025
9a251e9
import fix
michael-weigelt Feb 10, 2025
9c31252
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 10, 2025
ee7d23d
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 10, 2025
ae184f7
cost_call test
michael-weigelt Feb 10, 2025
c243ebd
failing test
michael-weigelt Feb 10, 2025
13acb57
simplify test
michael-weigelt Feb 10, 2025
8791dec
zero cost config
michael-weigelt Feb 11, 2025
38bd751
fix implementation of cost call
michael-weigelt Feb 12, 2025
50781a5
fix tests
michael-weigelt Feb 12, 2025
3619b79
fix implementation of cost call
michael-weigelt Feb 12, 2025
61d0176
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 12, 2025
9ba9fea
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 12, 2025
85f6870
fix benchmarks
michael-weigelt Feb 12, 2025
913ec93
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 12, 2025
2346715
invoke calls tests
michael-weigelt Feb 12, 2025
6efe416
curve exhaustiveness tests
michael-weigelt Feb 12, 2025
796b911
improved description
michael-weigelt Feb 12, 2025
e63566b
doc
michael-weigelt Feb 12, 2025
ec7d9c6
curve exhaustiveness tests
michael-weigelt Feb 12, 2025
0d4d8d0
Merge branch 'mwe/costs' into mwe/costs2
michael-weigelt Feb 12, 2025
870848a
Merge branch 'mwe/costs2' into mwe/costs3
michael-weigelt Feb 12, 2025
f00810f
ecdsa tests
michael-weigelt Feb 12, 2025
95b25c9
schnorr tests
michael-weigelt Feb 12, 2025
598d83e
vetkd tests
michael-weigelt Feb 12, 2025
ec9d723
Merge branch 'master' into mwe/costs
michael-weigelt Feb 12, 2025
e41dd00
clippy
michael-weigelt Feb 12, 2025
6fe4aaa
Update rs/interfaces/src/execution_environment.rs
michael-weigelt Feb 14, 2025
07ca8fa
comments
michael-weigelt Feb 14, 2025
9c82316
comments
michael-weigelt Feb 14, 2025
539868c
Merge remote-tracking branch 'origin' into mwe/costs
michael-weigelt Feb 14, 2025
c91d02e
saturate
michael-weigelt Feb 17, 2025
1dd00ec
Merge branch 'master' into mwe/costs
michael-weigelt Feb 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion rs/canister_sandbox/src/sandbox_server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ mod tests {
use ic_limits::SMALL_APP_SUBNET_MAX_SIZE;
use ic_logger::replica_logger::no_op_logger;
use ic_registry_subnet_type::SubnetType;
use ic_replicated_state::{Global, NumWasmPages, PageIndex, PageMap};
use ic_replicated_state::{Global, NetworkTopology, NumWasmPages, PageIndex, PageMap};
use ic_system_api::{
sandbox_safe_system_state::{CanisterStatusView, SandboxSafeSystemState},
ApiType, ExecutionParameters, InstructionLimits,
Expand Down Expand Up @@ -242,6 +242,7 @@ mod tests {
caller,
0,
IS_WASM64_EXECUTION,
NetworkTopology::default(),
)
}

Expand Down
26 changes: 26 additions & 0 deletions rs/config/src/subnet_config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,32 @@ impl CyclesAccountManagerConfig {
default_reserved_balance_limit: DEFAULT_RESERVED_BALANCE_LIMIT,
}
}

pub fn zero_cost(subnet_size: usize) -> Self {
Self {
reference_subnet_size: subnet_size,
canister_creation_fee: Cycles::zero(),
update_message_execution_fee: Cycles::zero(),
ten_update_instructions_execution_fee: Cycles::zero(),
ten_update_instructions_execution_fee_wasm64: Cycles::zero(),
xnet_call_fee: Cycles::zero(),
xnet_byte_transmission_fee: Cycles::zero(),
ingress_message_reception_fee: Cycles::zero(),
ingress_byte_reception_fee: Cycles::zero(),
gib_storage_per_second_fee: Cycles::zero(),
compute_percent_allocated_per_second_fee: Cycles::zero(),
duration_between_allocation_charges: Duration::from_secs(u64::MAX),
ecdsa_signature_fee: Cycles::zero(),
schnorr_signature_fee: Cycles::zero(),
vetkd_fee: Cycles::zero(),
http_request_linear_baseline_fee: Cycles::zero(),
http_request_quadratic_baseline_fee: Cycles::zero(),
http_request_per_byte_fee: Cycles::zero(),
http_response_per_byte_fee: Cycles::zero(),
max_storage_reservation_period: Duration::from_secs(u64::MAX),
default_reserved_balance_limit: Cycles::zero(),
}
}
}

/// If a component has at least one static configuration that is different for
Expand Down
51 changes: 42 additions & 9 deletions rs/cycles_account_manager/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -757,16 +757,15 @@ impl CyclesAccountManager {
reveal_top_up: bool,
) -> Result<Vec<(CyclesUseCase, Cycles)>, CanisterOutOfCyclesError> {
// The total amount charged consists of:
// - the fee to do the xnet call (request + response)
// - the fee to send the request (by size)
// - the fee for the largest possible response
// - the fee for executing the largest allowed response when it eventually arrives.
let transmission_fee = self.scale_cost(
self.config.xnet_call_fee
+ self.config.xnet_byte_transmission_fee * request.payload_size_bytes().get(),
// the fee to do the xnet call (request + response),
// the fee to send the request (by size),
// the fee for the largest possible response,
let transmission_fee = self.xnet_total_transmission_fee(
request.payload_size_bytes(),
subnet_size,
) + prepayment_for_response_transmission;

prepayment_for_response_transmission,
);
// and the fee for executing the largest allowed response when it eventually arrives.
let fee = transmission_fee + prepayment_for_response_execution;

self.withdraw_with_threshold(
Expand Down Expand Up @@ -797,6 +796,40 @@ impl CyclesAccountManager {
]))
}

/// The total amount for an xnet call transmission. Includes response transmission, but
/// excludes the response execution.
pub fn xnet_total_transmission_fee(
&self,
payload_size: NumBytes,
subnet_size: usize,
prepayment_for_response_transmission: Cycles,
) -> Cycles {
self.xnet_call_performed_fee(subnet_size)
+ self.xnet_call_bytes_transmitted_fee(payload_size, subnet_size)
+ prepayment_for_response_transmission
}

/// The total fee for an xnet call, including payload size, transmission (both ways)
/// and the reservation for the response execution. Corresponds to the amount of
/// cycles above the freezing threshold a canister must be for ic0.call_perform to
/// succeed.
pub fn xnet_call_total_fee(
&self,
payload_size: NumBytes,
execution_mode: WasmExecutionMode,
) -> Cycles {
let subnet_size = self.config.reference_subnet_size;
let prepayment_for_response_transmission =
self.prepayment_for_response_transmission(subnet_size);
let prepayment_for_response_execution =
self.prepayment_for_response_execution(subnet_size, execution_mode);
self.xnet_total_transmission_fee(
payload_size,
subnet_size,
prepayment_for_response_transmission,
) + prepayment_for_response_execution
}

/// Returns the amount of cycles required for executing the longest-running
/// response callback.
pub fn prepayment_for_response_execution(
Expand Down
60 changes: 60 additions & 0 deletions rs/embedders/src/wasm_utils/validation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -657,6 +657,66 @@ fn get_valid_system_apis_common(I: ValType) -> HashMap<String, HashMap<String, F
},
)],
),
(
"cost_call",
vec![(
API_VERSION_IC0,
FunctionSignature {
param_types: vec![ValType::I64, ValType::I64, I],
return_type: vec![],
},
)],
),
(
"cost_create_canister",
vec![(
API_VERSION_IC0,
FunctionSignature {
param_types: vec![I],
return_type: vec![],
},
)],
),
(
"cost_http_request",
vec![(
API_VERSION_IC0,
FunctionSignature {
param_types: vec![ValType::I64, ValType::I64, I],
return_type: vec![],
},
)],
),
(
"cost_sign_with_ecdsa",
vec![(
API_VERSION_IC0,
FunctionSignature {
param_types: vec![I, I, ValType::I32, I],
return_type: vec![ValType::I32],
},
)],
),
(
"cost_sign_with_schnorr",
vec![(
API_VERSION_IC0,
FunctionSignature {
param_types: vec![I, I, ValType::I32, I],
return_type: vec![ValType::I32],
},
)],
),
(
"cost_vetkd_derive_encrypted_key",
vec![(
API_VERSION_IC0,
FunctionSignature {
param_types: vec![I, I, ValType::I32, I],
return_type: vec![ValType::I32],
},
)],
),
];

valid_system_apis
Expand Down
94 changes: 94 additions & 0 deletions rs/embedders/src/wasmtime_embedder/system_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1248,6 +1248,100 @@ pub fn syscalls<
})
.unwrap();

linker
.func_wrap("ic0", "cost_call", {
move |mut caller: Caller<'_, StoreData>,
method_name_size: u64,
payload_size: u64,
dst: I| {
charge_for_cpu(&mut caller, overhead::COST_CALL)?;
with_memory_and_system_api(&mut caller, |s, memory| {
let dst: usize = dst.try_into().expect("Failed to convert I to usize");
s.ic0_cost_call(method_name_size, payload_size, dst, memory)
})
.map_err(|e| anyhow::Error::msg(format!("ic0_cost_call failed: {}", e)))
}
})
.unwrap();

linker
.func_wrap("ic0", "cost_create_canister", {
move |mut caller: Caller<'_, StoreData>, dst: I| {
charge_for_cpu(&mut caller, overhead::COST_CREATE_CANISTER)?;
with_memory_and_system_api(&mut caller, |s, memory| {
let dst: usize = dst.try_into().expect("Failed to convert I to usize");
s.ic0_cost_create_canister(dst, memory)
})
.map_err(|e| anyhow::Error::msg(format!("ic0_cost_create_canister failed: {}", e)))
}
})
.unwrap();

linker
.func_wrap("ic0", "cost_http_request", {
move |mut caller: Caller<'_, StoreData>,
request_size: u64,
max_res_bytes: u64,
dst: I| {
charge_for_cpu(&mut caller, overhead::COST_HTTP_REQUEST)?;
with_memory_and_system_api(&mut caller, |s, memory| {
let dst: usize = dst.try_into().expect("Failed to convert I to usize");
s.ic0_cost_http_request(request_size, max_res_bytes, dst, memory)
})
.map_err(|e| anyhow::Error::msg(format!("ic0_cost_http_request failed: {}", e)))
}
})
.unwrap();

linker
.func_wrap("ic0", "cost_sign_with_ecdsa", {
move |mut caller: Caller<'_, StoreData>, src: I, size: I, curve: u32, dst: I| {
let src: usize = src.try_into().expect("Failed to convert I to usize");
let size: usize = size.try_into().expect("Failed to convert I to usize");
charge_for_cpu_and_mem(&mut caller, overhead::COST_ECDSA, size)?;
with_memory_and_system_api(&mut caller, |s, memory| {
let dst: usize = dst.try_into().expect("Failed to convert I to usize");
s.ic0_cost_sign_with_ecdsa(src, size, curve, dst, memory)
})
.map_err(|e| anyhow::Error::msg(format!("ic0_cost_sign_with_ecdsa failed: {}", e)))
}
})
.unwrap();

linker
.func_wrap("ic0", "cost_sign_with_schnorr", {
move |mut caller: Caller<'_, StoreData>, src: I, size: I, algorithm: u32, dst: I| {
let src: usize = src.try_into().expect("Failed to convert I to usize");
let size: usize = size.try_into().expect("Failed to convert I to usize");
charge_for_cpu_and_mem(&mut caller, overhead::COST_SCHNORR, size)?;
with_memory_and_system_api(&mut caller, |s, memory| {
let dst: usize = dst.try_into().expect("Failed to convert I to usize");
s.ic0_cost_sign_with_schnorr(src, size, algorithm, dst, memory)
})
.map_err(|e| {
anyhow::Error::msg(format!("ic0_cost_sign_with_schnorr failed: {}", e))
})
}
})
.unwrap();

linker
.func_wrap("ic0", "cost_vetkd_derive_encrypted_key", {
move |mut caller: Caller<'_, StoreData>, src: I, size: I, curve: u32, dst: I| {
let src: usize = src.try_into().expect("Failed to convert I to usize");
let size: usize = size.try_into().expect("Failed to convert I to usize");
charge_for_cpu_and_mem(&mut caller, overhead::COST_VETKD, size)?;
with_memory_and_system_api(&mut caller, |s, memory| {
let dst: usize = dst.try_into().expect("Failed to convert I to usize");
s.ic0_cost_vetkd_derive_encrypted_key(src, size, curve, dst, memory)
})
.map_err(|e| {
anyhow::Error::msg(format!("ic0_cost_vetkd_derive_encrypted_key failed: {}", e))
})
}
})
.unwrap();

linker
.func_wrap("ic0", "call_with_best_effort_response", {
move |mut caller: Caller<'_, StoreData>, timeout_seconds: u32| {
Expand Down
6 changes: 6 additions & 0 deletions rs/embedders/src/wasmtime_embedder/system_api_complexity.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ pub mod overhead {
pub const CERTIFIED_DATA_SET: NumInstructions = NumInstructions::new(500);
pub const CONTROLLER_COPY: NumInstructions = NumInstructions::new(500);
pub const CONTROLLER_SIZE: NumInstructions = NumInstructions::new(500);
pub const COST_CALL: NumInstructions = NumInstructions::new(100);
pub const COST_CREATE_CANISTER: NumInstructions = NumInstructions::new(100);
pub const COST_HTTP_REQUEST: NumInstructions = NumInstructions::new(100);
pub const COST_ECDSA: NumInstructions = NumInstructions::new(100);
pub const COST_SCHNORR: NumInstructions = NumInstructions::new(100);
pub const COST_VETKD: NumInstructions = NumInstructions::new(100);
pub const DATA_CERTIFICATE_COPY: NumInstructions = NumInstructions::new(500);
pub const DATA_CERTIFICATE_PRESENT: NumInstructions = NumInstructions::new(500);
pub const DATA_CERTIFICATE_SIZE: NumInstructions = NumInstructions::new(500);
Expand Down
27 changes: 27 additions & 0 deletions rs/execution_environment/benches/lib/src/wat.rs
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,9 @@ pub struct Params2<P1, P2>(pub P1, pub P2);
/// System API call with 3 parameters.
pub struct Params3<P1, P2, P3>(pub P1, pub P2, pub P3);

/// System API call with 3 parameters.
pub struct Params4<P1, P2, P3, P4>(pub P1, pub P2, pub P3, pub P4);

/// Trait to render System API call parameters.
pub trait RenderParams {
/// Render System API call parameter import.
Expand Down Expand Up @@ -427,6 +430,30 @@ impl<P1: RenderParams, P2: RenderParams, P3: RenderParams> RenderParams for Para
}
}

/// Implement RenderParams trait for a System API call with 4 parameters.
impl<P1: RenderParams, P2: RenderParams, P3: RenderParams, P4: RenderParams> RenderParams
for Params4<P1, P2, P3, P4>
{
fn import(&self) -> String {
format!(
"{P1} {P2} {P3} {P4}",
P1 = self.0.import(),
P2 = self.1.import(),
P3 = self.2.import(),
P4 = self.3.import(),
)
}
fn call(&self) -> String {
format!(
"{P1} {P2} {P3} {P4}",
P1 = self.0.call(),
P2 = self.1.call(),
P3 = self.2.call(),
P4 = self.3.call()
)
}
}

/// System API call result.
/// The dead code is allowed as this common module is used across a few benchmark binaries.
#[allow(dead_code)]
Expand Down
Loading